This document provides an overview of the data streaming ecosystem at Booking.com. It discusses how Booking.com uses Apache Kafka, Kafka Connect, and related tools across over 300 clusters containing over 350 brokers to handle large volumes of streaming data from its various services and applications. Key aspects of Booking.com's data streaming infrastructure are highlighted, including its use of multiple data centers, global and local clusters, monitoring and alerting systems, and operational best practices.

1. Data Streaming Ecosystem
Alex Mironov | 10/17/2018

2. intro. who am I?

3. what is ?

4. ● travel E-Commerce, part of
Booking Holdings (NASDAQ: BKNG)
● 28,000,000+ listings of hotels, homes,
apartments and other places to stay
● 140,000+ destinations in 230 countries
● 1,550,000+ rooms nights reserved daily
● based in Amsterdam, with 198 offices
worldwide

5. data streaming ecosystem
services
kafka
streams
schema
registry
connect

6. Personalization
Notifications
Experiment Tracking
Payments
Events
Logging
Security

7. 30+ clusters

8. 300TB of data

9. 350+
brokers
1.5k+
topics
12k+
partitions
2.5M
msgs/sec
4.5
GBps in
4.7
GBps out

10. baremetal core
satellite containers

11. ● 24 cores
● 128GB memory
● 2x1.6TB SSD in RAID 0
● or 8TB iSCSI, NetApp E-Series
● 10G network
● CentOS 7
● puppet

12. set of small clusters
instead of a few giant ones

13. ● vanilla kafka 1.0.1
● 8 brokers per cluster
● rack awareness
● auto.create.topics.enable = false
● tuning
○ replica fetchers
○ replica fetch max bytes
○ network and IO threads
○ socket buffers for 10G net

14. ● N partitions where N is multiple of number of
brokers in a cluster
● retention = 1 hour
● replication factor = 3 (rack-aware)
● min insync replicas = 1
● 1MB max message size
● 1 GB segments
● timestamp = CreateTime

16. DC 1

17. zookeeper
DC 1

18. zookeeper
kafka
DC 1

19. DC 1 DC 2 DC 3
zookeeper
kafka
zookeeper
kafka
zookeeper
kafka

20. kafka
DC 1 DC 2 DC 3
kafka kafka
producers
consumers
producers
consumers
producers
consumers

21. kafka
DC 1 DC 2 DC 3
kafka kafka
producers
consumers
producers
consumers
producers
consumers

22. zookeeper
kafka
zookeeper
kafka
zookeeper
kafka
DC 1 DC 2 DC 3

23. zookeeper
kafka
global zookeeper
global kafka
zookeeper
kafka
zookeeper
kafka
DC 1 DC 2 DC 3

24. zookeeper
kafka
global zookeeper
global schema registry
global kafka
zookeeper
kafka
zookeeper
kafka
DC 1 DC 2 DC 3

25. zookeeper
kafka
global zookeeper
global schema registry
global kafka
zookeeper
kafka
zookeeper
kafka
DC 1 DC 2 DC 3
kafka monitor kafka connect replicator

26. zookeeper
kafka
global zookeeper
global schema registry
global kafka
zookeeper
kafka
zookeeper
kafka
kafka monitor kafka connect replicator
kontrole
DC 1 DC 2 DC 3

27. zookeeper
kafka
global zookeeper
global schema registry
global kafka
zookeeper
kafka
zookeeper
kafka
kafka monitor kafka connect replicator
kontrole
DC 1 DC 2 DC 3

29. ● dropwizard + react
● topic ownership & attribution
● multi-cluster topic configuration
and provisioning
● topic discovery
● schema management (soon)
● connector & replicator configuration (soon)

30. zookeeper
kafka
global zookeeper
schema registry
global kafka
zookeeper
kafka
zookeeper
kafka
kafka monitor kafka connect replicator
kontrole
DC 1 DC 2 DC 3

31. replicator
kafka
PROD
DEV
kafka

32. replicator
kafka
PROD
DEV
kafka
sink connector

33. sampled traffic (e.g. 5%)
DEV
replicator
kafka
kafka
PROD
sink connector

34. zookeeper
kafka
global zookeeper
schema registry
global kafka
zookeeper
kafka
zookeeper
kafka
kafka monitor kafka connect replicator
kontrole
DC 1 DC 2 DC 3

35. distributed mode

36. distributed mode

37. standalone mode

38. landscaper
+ + =

40. repo
push

41. Gitlab
CI/CD
repo
push

42. Gitlab
CI/CD
landscaper
repo
push
replicator kafka connect kafka monitor

43. Gitlab
CI/CD
landscaper
repo
push
tiller
replicator kafka connect kafka monitor

44. monitoring

45. jmx jmxtrans graphite

47. SLIs & SLOs

49. zookeeper
kafka
global zookeeper
schema registry
global kafka
zookeeper
kafka
zookeeper
kafka
kafka monitor kafka connect replicator
kontrole
DC 1 DC 2 DC 3

51. kafka monitor
kafka

52. kafka monitor
produce every 100ms
with seq id and track
availability
kafka

53. kafka monitor
produce every 100ms
with seq id and track
availability
consume and track
pipeline latency
kafka

54. kafka monitor
kafka
produce every 100ms
with seq id and track
availability
consume and track
pipeline latency

55. alerting

56. ● under-replicated partitions
● ISR churn
● failed requests
● offline partitions
● controller discrepancy
● open FDs
● disk space
● ZK backups
● SLO violation

57. bosun

58. alert kafka_underreplicated {
template = kafka_underreplicated
$last_minutes = 60
$warn_minutes_threshold = 30
$crit_minutes_threshold = 60
$num_underreplicated = graphite(...)
$fail_minutes = streak($num_underreplicated > 0)
crit = ($fail_minutes >= $crit_minutes_threshold)
warn = ($fail_minutes >= $warn_minutes_threshold)
}

59. not every alert is critical

60. streaks instead of spikes

61. war stories

62. Curious case
of missing space

64. ~ ./kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000526818225.log | head
Dumping 00000000000526818225.log
Starting offset: 526818225
baseOffset: 526818225 lastOffset: 526818225 baseSequence: -1 lastSequence: -1 producerId:
-1 producerEpoch: -1 partitionLeaderEpoch: 167 isTransactional: false position: 0
CreateTime: 1636572835 isvalid: true …
baseOffset: 526818226 lastOffset: 526818226 baseSequence: -1 lastSequence: -1 producerId:
-1 producerEpoch: -1 partitionLeaderEpoch: 167 isTransactional: false position: 420
CreateTime: -1 isvalid: true …
baseOffset: 526818227 lastOffset: 526818227 baseSequence: -1 lastSequence: -1 producerId:
-1 producerEpoch: -1 partitionLeaderEpoch: 167 isTransactional: false position: 624
CreateTime: -1 isvalid: true …

65. ~ ./kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000526818225.log | head
Dumping 00000000000526818225.log
Starting offset: 526818225
baseOffset: 526818225 lastOffset: 526818225 baseSequence: -1 lastSequence: -1 producerId:
-1 producerEpoch: -1 partitionLeaderEpoch: 167 isTransactional: false position: 0
CreateTime: 1636572835 isvalid: true size: 420 magic: 2 compresscodec: NONE crc:
1140091055
baseOffset: 526818226 lastOffset: 526818226 baseSequence: -1 lastSequence: -1 producerId:
-1 producerEpoch: -1 partitionLeaderEpoch: 167 isTransactional: false position: 420
CreateTime: -1 isvalid: true size: 204 magic: 2 compresscodec: NONE crc: 4150695476
baseOffset: 526818227 lastOffset: 526818227 baseSequence: -1 lastSequence: -1 producerId:
-1 producerEpoch: -1 partitionLeaderEpoch: 167 isTransactional: false position: 624
CreateTime: -1 isvalid: true size: 9129 magic: 2 compresscodec: NONE crc: 426720461

66. ~ ./kafka-run-class.sh kafka.tools.DumpLogSegments --files 00000000000526818225.log | head
Dumping 00000000000526818225.log
Starting offset: 526818225
baseOffset: 526818225 lastOffset: 526818225 baseSequence: -1 lastSequence: -1 producerId:
-1 producerEpoch: -1 partitionLeaderEpoch: 167 isTransactional: false position: 0
CreateTime: 1636572835 isvalid: true size: 420 magic: 2 compresscodec: NONE crc:
1140091055
baseOffset: 526818226 lastOffset: 526818226 baseSequence: -1 lastSequence: -1 producerId:
-1 producerEpoch: -1 partitionLeaderEpoch: 167 isTransactional: false position: 420
CreateTime: -1 isvalid: true size: 204 magic: 2 compresscodec: NONE crc: 4150695476
baseOffset: 526818227 lastOffset: 526818227 baseSequence: -1 lastSequence: -1 producerId:
-1 producerEpoch: -1 partitionLeaderEpoch: 167 isTransactional: false position: 624
CreateTime: -1 isvalid: true size: 9129 magic: 2 compresscodec: NONE crc: 426720461
10th November 2021

67. log.message.timestamp.difference.max.ms

68. Down-conversion
is your enemy

69. 150 MB/s IO

70. 500 MB/s IO

73. num.io.threads

74. keep thou clients fresh

75. Thank you!
All references to “Booking.com", including any mention of “us”, “we” and “our” refer to Booking.com BV, the company behind Booking.com™