"Apache Kafka is Robinhood’s most mission-critical infrastructure. It is used in every line of Robinhood’s business, from stock and crypto trading to self-clearing, market data and data science. Most of our microservices are written in either Golang or Python, and we have multiple librdkafka-based client library wrappers (kafkahood) developed in both languages. As Robinhood grows, it has become challenging for infrastructure engineers to manage the different requirements that the multitude of application teams have for their producers and consumers. To efficiently manage all Kafka consumers at Robinhood, we have developed a Consumer Proxy that decouples the following: ● Kafka consumer logic from message processing in application business logic ● Resource utilization in Kafka consumer from that in application container ● Kafka consumer failure from application failure The Consumer Proxy is built in a Kubernetes sidecar container, it offers stateless processing, and it manages commits and timeouts. Most importantly, applications need the ability to scale up during market hours and scale down after extended trading hours, and the Consumer Proxy will allow them to easily rebalance consumer groups during these scaling events. Eventually, specific language library wrappers can remain a """"thin"""" layer whose complexities are only to facilitate the business logic of the corresponding application team with what they should do with the payload. Unifying Kafka consumers and producers for an application team in one sidecar proxy container allows the infrastructure team to effectively manage clients at Robinhood. If your organization is looking to centralize Kafka consumption logic to a singular client library (instead of multiple different client libraries), please attend this talk to see how Robinhood does it so that the infrastructure team can focus development on a singular library rather than N different client libraries."

1. © 2023 Robinhood Markets, Inc. 1
Kafkaproxy at Robinhood
MunYong Jang
Tony Chen
Wenlong Xiong
Sreeram Ramji
Chandra Kuchi
“Robinhood” and the Robinhood feather logo are registered trademarks of Robinhood Markets, Inc.
All other names are trademarks of and/or registered trademarks of their respective owners.

2. © 2023 Robinhood Markets, Inc. 2
1. Background
2. Why
3. Inner Workings
4. Challenges
5. Future
Agenda

3. © 2023 Robinhood Markets, Inc.
Background on RH
- All services run on k8s (mostly python/golang)
- Asynchronous service <-> service communication
- all material services use Kafka
- All pricing information flow through Kafka
- All orders flow through Kafka
- Eliminated Kafka as SPOF for the critical order path
- “Sharded” kafka set up
- Each cluster with racks in 3 separate AZs

4. © 2023 Robinhood Markets, Inc.
What is Kafkaproxy?
- Sidecar container
- Houses the kafka consumers
- Responsible for just the consumption of the kafka messages from the brokers
- Relays the kafka messages to the application container over gRPC
- Written in Java – using the open source Apache Kafka Clients

5. © 2023 Robinhood Markets, Inc.
Pre-Kafkaproxy World

6. © 2023 Robinhood Markets, Inc.
Post-Kafkaproxy World

7. © 2023 Robinhood Markets, Inc.
But why?

8. © 2023 Robinhood Markets, Inc.
But why?
- Centralization of client logic into a single library rather than two

9. © 2023 Robinhood Markets, Inc.
- Centralization of client logic into a single library rather than two
- Control over the roll-out process of library version bumps
But why?

10. © 2023 Robinhood Markets, Inc.
- Centralization of client logic into a single library rather than two
- Control over the roll-out process of library version bumps
- Exact telemetry into resource utilization devoted to the consumers
But why?

11. © 2023 Robinhood Markets, Inc.
- Centralization of client logic into a single library rather than two
- Control over the roll-out process of library version bumps
- Exact telemetry into resource utilization devoted to the consumers
- Ability to use whichever language is best suited
But why?

12. © 2023 Robinhood Markets, Inc.
A few hurdles to overcome...

13. © 2023 Robinhood Markets, Inc.
A few hurdles to overcome...
- Container lifecycle maintenance

14. © 2023 Robinhood Markets, Inc.
A few hurdles to overcome...
- Container lifecycle maintenance
- C vs Java performance (with gRPC overhead?)

15. © 2023 Robinhood Markets, Inc.
A few hurdles to overcome...
- Container lifecycle maintenance
- C vs Java performance (with gRPC overhead?)
- Migration

16. © 2023 Robinhood Markets, Inc. 16
Inner Workings
● Application container to sidecar container requests:
○ RegisterConsumer
○ StartConsuming
○ DeregisterConsumer
● Sidecar container to application container requests:
○ HandleMessage

17. © 2023 Robinhood Markets, Inc. 17
Problem 1: Containers
Pod
Application
Container
Javaproxy
Container
Pod
Application
Container
Javaproxy
Container

18. © 2023 Robinhood Markets, Inc.
Health Check
● A unidirectional health check from the sidecar container to the application container
○ This health check mechanism exists for each unique consumer registered by the application container
● HandleMessage requests from sidecar container to application container is crucial in this health check mechanism

19. © 2023 Robinhood Markets, Inc.

20. © 2023 Robinhood Markets, Inc.

21. © 2023 Robinhood Markets, Inc.
Problem 2: Java vs C?

22. © 2023 Robinhood Markets, Inc.
Problem 3: Migration
type Consumer interface {
Start() error
Stop()
Pause(partitions []core.TopicPartition) error
Resume(partitions []core.TopicPartition) error
}
type Callback interface {
OnConsume(CallbackContext, []*core.Message)
OnError(context.Context, core.ClientError)
}

23. © 2023 Robinhood Markets, Inc.
Forward looking
- Centralized DLQ implementation
- Unified telemetry/tracing across different languages
- Centralized multicluster consumption
- Producer-side

24. © 2023 Robinhood Markets, Inc.
Takeaways
- Fail-fast to determine if the project would be successful
- Make migrations as seamless as possible for application teams
- Enable Streaming Platform team to easily support new language clients

25. © 2023 Robinhood Markets, Inc.
Acknowledgements
- Wenlong Xiong
- Sreeram Ramji
- Shaan Appel
- Chandra Kuchi
- Vijay Kistampalli (formerly RH, now at Striim)
- Mia Liu (formerly RH, now at Snap)