Robinhood uses Kafka in every line of its business, from stock and crypto trading to clearing and data analytics. One interesting aspect of our architecture is that many of our microservices leveraging Kafka are written in Python. When you combine Python's relatively slow performance coupled, its reliance on process-based parallelism and Robinhood’s scale, the result is a massive fleet of application processes producing to and consuming from our Kafka clusters. This fleet generates an atypical workload on Kafka that warrants a deeper investment in scalability and reliability. This talk discusses our investments in Kafka infrastructure for a large-scale Python-based environment: kafkahood: our librdkafka-based client library wrapper that codifies best practices, sane defaults and deep client-side observability. kafkaproxy: a Rust-based sidecar proxy that reduces connection fan-in from Python gunicorn worker pools to our Kafka clusters. We'll also present challenges we encountered along the way and share our learnings with the audience.

1. Taming a Massive Fleet of Python-based
Kafka Apps at Robinhood

2. @nickdella
Staff Software Engineer, Streaming Platform
@ckkuchi
Engineering Manager, Streaming Platform

3. 3

4. 4
Stock trading Clearing Crypto trading
Messaging and push
notifications
Kafka use cases at Robinhood

6. kafkahood
Taming non-conforming and potentially dangerous application patterns via
Robinhood’s internal wrapper library for Python and Golang Kafka clients

7. Why wrap?

8. Library upgrades

9. Standardize configurations and defaults

10. Detailed client metrics for free

11. 11
Error handling and logging Dead letter queue Message serialization
More reasons to wrap
boom!

12. 12
Roll out library changes via feature flags
Icons via icons8.com
servars/kafkahood/enable_auth:
{"roles": ["sheriff-server"]}

13. 13
Roll out library changes via feature flags
Icons via icons8.com
servars/kafkahood/enable_auth:
{"roles": ["sheriff-server, sickle-daemons"]}

14. 14
Roll out library changes via feature flags
Icons via icons8.com
boom! boom!
servars/kafkahood/enable_auth:
{"roles": ["sheriff-server, sickle-daemons"]}

15. 15
Rollback!
Icons via icons8.com
servars/kafkahood/enable_auth:
{"roles": []}

16. 16
Clusters Brokers
6 160
Topics/partitions
1400 topics
90k partitions
Messages/second
(all clusters)
API requests/sec
(largest cluster)
2.2M 500k
Inbound client connections
(largest cluster)
> 100k

17. 17
Clusters Brokers
6 160
Topics/partitions
1400 topics
90k partitions
Messages/second
(all clusters)
API requests/sec
(largest cluster)
2.2M 500k
Inbound client connections
(largest cluster)
> 100k !

18. Scaling Python-based Kafka apps

22. kafkaproxy
Taming a massive fleet of Python Kafka clients via a sidecar proxy written in Rust

23. Requirement(s)

26. Sidecar as a solution

27. Why Rust
● Efficiency in terms of memory and cpu footprint
● Previous experience with librdkafka
● Constrained system with less LoC
● Librdkafka library ticked lot of boxes
● Functional syntax

28. Effect of proxy on number of connections

29. Lessons learnt on rollout
● Rolling sidecar onto kubernetes apps
○ Use feature flags for quick rollback
○ Monitor latency and throughput
● Getting configuration right for various apps
● Alpine and DNS issues
● Kubernetes sidecar ordering issues on pod
bootup

30. Future Work
● GRPC support for bidirectional streams for callbacks
● Daemonsets as a fallback to sidecars
● (Maybe) Consumer proxy

31. ThankYou