In this presentation, I will talk about my firsthand experience dealing with the unique challenges of running Kafka at a massive scale. If you ever thought that running Kafka is difficult, this talk may change your mind and provide you with valuable insights into how to configure a Kafka cluster efficiently, how to manage Kafka for enterprise customers and how to measure, monitor and maintain the Quality of Kafka Service. Our production Kafka cluster runs over 1500+ VMs, and serves over 10 GBPS data spread across hundreds of topics for multiple teams across Microsoft. We built a self-serve Kafka management service to make the process manageable and scalable across many teams. In this talk, I will also share insights about running Kafka in Private vs multi-tenant mode, supporting failover and disaster recovery requirements, and how to make Kafka Compliant with regulatory certifications such as ISO, SOC, FEDRAMP, etc. Presented by Nitin Kumar, Microsoft Systems Track

1. Kafka Reliability Guarantees
@ 1 Trillion Events/day
Nitin Kumar

2. What is Microsoft’s mission?
To empower every person and
every organization on the
planet to achieve more.
Xbox 55 million active users
O365 100 million active users
Skype 300 million active users
LinkedIn 500 million active users
Bing 873 million searches per day

3. Why do we care about NRT?
• Fast BI – Near Real time customer-facing Bing Ads campaign
reports
• Bing live site telemetry
• Measures Bing Quality of Service
• Office 365 – Near real time insights
• Actionable insights on customers’ experience,
engagement and satisfaction
• Security real time detection and response
• Skype telemetry
• LinkedIn

4. Devices
Devices
NRT Processing
Service 1
Internal Data
Warehouse
Service 2
Service 3
Monitoring and
Alerting
CassandraKafka
Devices
Kafka
SQL Data
Mart
Operational
Intelligence
Analytics
DB
Interactive
Analytics
USQL Streaming
Spark
Gateway
Azure
Data Lake
Event
Hubs
Azure
Storage
Stream Analytics

5. NRT pipelines
Client SDK
Collector
Siphon
connector
API
Management
UI
Metadata

6. 1 trillion (8 million per sec)
EVENTS PER DAY PEAK INGRESS
800 TB (10 GB per Sec)
INGRESS PER DAY
2,000
PRODUCTION KAFKA BROKERS (Kafka 0.10)
20 (D14, 10x1 TB disks)
CLUSTERS
500
TOPICS
500
CONNECTORS
Enterprise Data Bus
Near real-time
Compliant
No data dead-ends
Hyper scale
Reliable
Network effects
Kafka @ Microsoft

7. Running Kafka at scale?
73%
21%
6%
Operating Kafka Experience
Difficult
Easy
I don’t know

8. Life is hard – No pain, no gain

9. Light at the end of tunnel

10. Little things make big things happen

11. Three little things
Centralized
metrics and
logs
Self
serve
Automate
Cluster
creation

12. 1. Automate Cluster Creation

13. HDInsight
Azure Kafka
cluster
creation

14. Defining
Tenant Model
Principals
• Administrator
• Users
Resources
• Topics
• Connectors
Quota
• Storage
• Throughput

15. Tenancy Model: Single Tenant
Tenant A
Tenant B
Tenant C
Multiple instances
Single tenant per instance

16. Tenancy Model: Multi Tenant
Single instance
Multiple tenant per instance
Tenant B
Tenant C
Tenant A

17. Tenancy Model: Multi Tenant, Multi Instance
Multiple instances
Multiple tenant per instance
Tenant B
Tenant C
Tenant A

18. Siphon Tenant Architecture
Siphon Private Cluster
Siphon Shared Cluster
Tenant 3 Endpoint3
Tenant 2 Endpoint2
Deployment unit 2
Collector Connector
Deployment unit 3
Collector Connector
Tenant 1 Endpoint1 Deployment unit 1
Collector Connector
Tenant Endpoint
Deployment unit
Collector Connector

19. 2. Self Serve

20. Self Serve
Kafka
Management
portal
• Tenant management
• Topic management
• Connector management

21. S e l f s e r v e K a f k a – T e n a n t
M a n a g e m e n t

22. S e l f s e r v e K a f k a – T o p i c
M a n a g e m e n t

23. S e l f s e r v e K a f k a – T o p i c
M a n a g e m e n t

24. S e l f s e r v e K a f k a – C o n n e c t o r
M a n a g e m e n t

25. 3. Centralized metrics and logs

26. Topic View

27. Broker view

28. Broker view

29. Connector view

30. Connector performance view

32. Cluster Quality of Service (QoS)

33. Maintaining 999 SLA
1) Metrics drop creates DRI tickets
2) Postmortem of tickets
3) Weekly review of customer impacting DRI tickets

34. Harder problems – Pet Peeves
• Data skew in disks
• Throttling per Partition
• Controller issues
• Compliance
• Encryption at rest
• Locked down cluster

35. Three thinking hats
Think Instrumentation first
• Single SDK
• Single schema
Think collection next
• Single Databus Pub/Sub system (Kafka, Event Hub, Kinesis etc.)
• Publish once, subscribe many times
Think processing
• No one size fits all – Spark, Kafka streams, Flink, etc.

36. Disk 1
Disk 2
Disk 3
(added new)
T1-P1
T2-P2 T2-P3
T1-P2 T2-P1
* Disk-3 is a newly added disk and we could move some partitions there.
* If Topic T1 is a higher IO demanding topic, then prefer to put T1-P1 and T1-P2 on separate drives.
Imbalance inside a Broker disks
Disk 4
(went offline)

37. Kafka Encryption
Azure KeyVault with ACL and
restricted access
Kafka
Producer
1) Fetches the
latest version of
the AES Key and
uses it to
encrypt
2) Then, it
sends the
encrypted data
to Kafka to
store at rest.
Consumer
1) Fetch
encrypted data
from Kafka
2) Fetch Key
from KeyVault
3) Decrypt
messages
using the key
Message Format
<FormatId>
<Salt>
<4 bytes: length of next field>
<KeyVaut URL of the encryption Key>
<4 bytes: length of next field>
<encrypted byte[] of customer data>