This is a story about what happens when a distributed system becomes a big part of a small team's infrastructure. This distributed system was Kafka and the team size was one engineer. I will discuss my failures along with my journey of deploying Kafka at scale with very little prior distributed systems experience. In this presentation, we will discuss how unique insights in the following organization culture, engineering and metrics created tailwinds and headwinds. This presentation will be a tactical approach to conquering a complex system with an understaffed team while your business is growing fast. I will discuss how the use case and resilience requirements for our Kafka cluster change as the user base grew from 100K users to over 6 million.

1. Tackling Kafka with a Small Team

2. @JarenGlover

5. Streaming Evolution at Robinhood
+
Technical Trade-offs

7. circa 2015

8. Product Analytics

9. We wanted to build our own event pipeline.

10. Kafka Concerns

13. Real Time Business Decisions

14. Run Experiments

15. Actual Application Visibility

16. Is Kafka Down?

17. Application Engineer’s Perspective

18. Application Engineer’s Perspective
● Implicit Trust

19. Application Engineer’s Perspective
● Implicit Trust
● Retry Logic

20. Application Engineer’s Perspective
● Implicit Trust
● Retry Logic
● Library Ownership

21. Infrastructure Engineer’s Perspective

25. Infrastructure Engineer’s Perspective
● Suffering from Success
● Technical Whack-A-Mole
● Library Ownership

26. Engineering Trade Offs

27. What We Got Right
Replacing Unhealthy Brokers

30. … Recently Got Right
Tooling for Rolling Restarts

32. Thanks Prometheus and Fabric

33. Soon to be Right ...
Overhaul the Client Library

34. Shared Library Improvements
● More Implementation Structure

35. Shared Library Improvements
● More Implementation Structure
● Observability for Free

36. Shared Library Improvements
● More Implementation Structure
● Observability for Free
● Integration and Regression Testing

37. “You can do it too!”