Apache Kafka is a key part of the Big Data infrastructure at Salesforce, enabling publish/subscribe and data transport in near real-time at enterprise scale handling trillions of messages per day. In this session, hear from the teams at Salesforce that manage Kafka as a service, running over a hundred clusters across on-premise and public cloud environments with over 99.9% availability. Hear about best practices and innovations, including: * How to manage multi-tenant clusters in a hybrid environment * High volume data pipelines with Mirus replicating data to Kafka and blob storage * Kafka Fault Injection Framework built on Trogdor and Kibosh * Automated recovery without data loss * Using Envoy as an SNI-routing Kafka gateway We hope the audience will have practical takeaways for building, deploying, operating, and managing Kafka at scale in the enterprise.

1. Tales from the
four-comma club
Managing Kafka as a Service at
Salesforce
Paul Davidson, Software Architect
Lei Ye, Principal Software Engineer

2. Paul Davidson
Software Engineering Architect Logo

3. Lei Ye
Principal Software Engineer

4. What we will talk about
Operating Kafka at Global Scale
Many brokers, Many clusters, Many workloads, Many substrates, Many tenants
Connectivity
Global Data Pipelines
Testing
Automating for Availability

5. Salesforce Kafka as a Service
Built on open-source Apache Kafka
Managed Solution for
Pub/Sub and Data Pipelines at Salesforce

6. Salesforce Kafka
Operational and Customer-Facing
Pub / Sub
Including multi-tenant (100s of thousands)
Salesforce Platform Events
Change Data Capture
Capture changes to Salesforce Data Records as events
Use Cases

7. Salesforce Kafka
Stream Processing
Spark, Flink, etc.
Cross-site Data Pipelines
Storage and Aggregation (e.g. Logs, Metrics, Traces, other)
Use Cases

8. Salesforce Kafka
100+ Kafka clusters
Kafka 2.6
50,000+ topics, 300K+ partitions
2500+ brokers, 40+ PB storage
Largest clusters 250+ brokers
3+ Trillion Events/Day (average)
Doubling each year
Current Scale

9. Management and Automation Tooling
● Investing in orchestration and self-healing
○ Kubernetes in public cloud
○ Choria for on-prem deployments and self-healing
○ Continuous automated patching
○ Further growth requires more automation!
● Tooling
○ Cruise Control, Xinfra Monitor, Argus (internal OpenTSDB based service), Splunk
○ Topic Management and visibility: in-house tools (Wrangler - cross-cluster UI)

11. Kafka API Gateway
● Native Kafka Endpoint for cross-site connectivity
● Standard load balancer + Envoy as a Layer 4
SNI-routing Proxy
● Envoy routes TLS connection to correct broker
using SNI (Server Name Indication)
○ SNI = TLS Extension supported by Kafka Clients
● Allows secure cross-site connectivity with
standard Kafka Clients

12. Data Pipelines
● Mirus: Reliable, high performance replication tool
○ Mirus Kafka Connector
○ Open Source: https://github.com/salesforce/mirus
○ Low-latency replication
● Object Store Connector
○ Support S3, GCS, etc.
○ High-throughput replication
● Provides global cross-site data transport
Replicating Events between Data Centers

13. Kafka Operation Experience

14. Case Study
● Multiple dedicated regional Kafka clusters on Public Cloud to stream security log data to
Defensive Security data lakes
● Requirements
○ High availability
○ Durability
○ Scalability
○ Low operation cost
● We learnt a lot of Kafka operation experience
○ Offline/Under-replicated partition
○ Frozen broker
○ Disk failure
○ Patching noise
● We built useful tools to improve Kafka testing and operation
○ Kafka fault injection testing infrastructure
○ Kafka auto-fix ops tool to enable automated partition and broker recovery without data loss

15. Kafka Testing Infrastructure
● Reliable and easy-to-configure to run multiple types of testings
○ functional/performance/load/scale/upgrade-and-downgrade/patching tests
● Deterministic, reproducible infrastructure to repeat all types of testing
● Precise fault-injection capability to trigger fault events to simulate production incident
● Orchestrating all components to provide a high quality of QA for Kafka

16. Kafka Testing Infrastructure Architecture
● Test Spec -- a JSON format test file
● Test Coordinator -- the control plane of
testing infrastructure. It’s a web service
parsing the test spec and submitting a
test to the Kafka cluster
● Load Generator -- simulates producer and
consumer traffic over Kafka topics
● Ops Tool -- auxiliary tools to manage and
configure the Kafka cluster
● Fault Injection Framework -- based on
Trogdor and Kibosh to simulate the
network/process/filesystem faults
● Result Analysis -- tracks log and metrics
from Kafka cluster to analyze testing
results

17. Test Spec Format

18. Running Test Spec
Default Kafka Cluster Configuration: Min-ISR = 2; Produce Ack = ALL; Unclean leader election = false;
Inject network fault [7], [1, 3, 5]
00:00 00:05
Generate producing and consuming load to test1_large
Inject network fault [7], [2, 4, 6]
Inject disk fault on the partition storage of [7]
00:10 00:15 00:20
00:00 Broker 7 starts with a healthy state:
Topic: test1_large Partition: 0 Leader: 7 Replicas: 7,1,2 Isr: 7,1,2
Topic: test1_large Partition: 1 Leader: 7 Replicas: 7,3,4 Isr: 7,4,3
Topic: test1_large Partition: 2 Leader: 7 Replicas: 7,3,4 Isr: 7,4,3
Topic: test1_large Partition: 3 Leader: 7 Replicas: 7,5,6 Isr: 7,6,5
Topic: test1_large Partition: 4 Leader: 7 Replicas: 7,5,6 Isr: 7,6,5
00:05 Broker 7 starts degrading, broker 1,3,5 fall out of ISR:
Topic: test1_large Partition: 0 Leader: 7 Replicas: 7,1,2 Isr: 7,2
Topic: test1_large Partition: 1 Leader: 7 Replicas: 7,3,4 Isr: 7,4
Topic: test1_large Partition: 2 Leader: 7 Replicas: 7,3,4 Isr: 7,4
Topic: test1_large Partition: 3 Leader: 7 Replicas: 7,5,6 Isr: 7,6
Topic: test1_large Partition: 4 Leader: 7 Replicas: 7,5,6 Isr: 7,6
00:10 Broker 7 continues to degrade, broker 2,4,6 fall out of ISR:
Topic: test1_large Partition: 0 Leader: 7 Replicas: 7,1,2 Isr: 7
Topic: test1_large Partition: 1 Leader: 7 Replicas: 7,3,4 Isr: 7
Topic: test1_large Partition: 2 Leader: 7 Replicas: 7,3,4 Isr: 7
Topic: test1_large Partition: 3 Leader: 7 Replicas: 7,5,6 Isr: 7
Topic: test1_large Partition: 4 Leader: 7 Replicas: 7,5,6 Isr: 7
00:15 Broker 7 fails due to disk failure, all five partitions go offline:
Topic: test1_large Partition: 0 Leader: -1 Replicas: 7,1,2 Isr: 7
Topic: test1_large Partition: 1 Leader: -1 Replicas: 7,3,4 Isr: 7
Topic: test1_large Partition: 2 Leader: -1 Replicas: 7,3,4 Isr: 7
Topic: test1_large Partition: 3 Leader: -1 Replicas: 7,5,6 Isr: 7
Topic: test1_large Partition: 4 Leader: -1 Replicas: 7,5,6 Isr: 7

19. How to Recover?
● For offline partitions, it should perform the following steps
○ Identify the most advanced replica that has the highest LogEndOffset using jmxtool
./bin/kafka-run-class.sh kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://127.0.0.1:9010/jmxrmi --object-name
kafka.log:type=Log,name=LogEndOffset,topic=test1_large,partition=0 --one-time true
○ Elect it as a new leader by using partition reassignment
■ If new leader is the first in partition replica assignment, it can skip reassignment step
■ Execute Kafka partition reassignment tool
./bin/kafka-reassign-partitions.sh --reassignment-json-file partitions-to-move.json --execute
■ Make sure the new leader is the first live replica in the partition replica assignment
○ Enable the unclean leader election for the topic
○ Disable the unclean leader election for the topic after recovery
● For bad brokers, it needs to decommission the hosts, re-provision and add the new hosts
back to the fleet

20. Kafka Auto-fix Ops Tool
● One-stop tool to automate the entire partition and broker recovery process
● Short MTTR (~2 mins)
● No data loss

21. Conclusion

22. Summary
Salesforce relies on Kafka at global scale
Global data transport
Widely varied workloads
Massive multi-tenancy
Availability at Scale requires investing in
Automated Testing
Automated Operations and Self-healing
Visibility, Monitoring, and Alerting
More growth requires more automation!

23. Thank You