This document summarizes Slack's use of Druid, an open-source analytics database, to measure performance of its API. It discusses Slack's growth from 5 employees and 350,000 active users in 2016 to 800+ employees and 6 million active users in 2017. It describes how Druid is used to store and query 500+ tables and handle 400,000 accesses to Slack's data warehouse per day. The document outlines Druid's infrastructure, including its use of Kafka for a near real-time pipeline and autoscale of historical and broker nodes. Finally, it discusses challenges faced like cascading failures, optimizing forward index fields, supporting SQL, and bridging batch and real-time tables.

1. Ananth Packkildurai
November 28, 2017
1
Measuring Slack API performance using Druid

2. Public launch: 2014 800+ employees across
7 countries worldwide
HQ in San Francisco
Diverse set of industries
including software/technology, retail, media,
telecom and professional services.
About Slack

3. An unprecedented adoption rate

4. Agenda
1. A bit history.
2. Druid infrastructure & usecases
3. Challenges.

5. A bit history

6. March 2016
5 350+ 2M
Data Engineers Slack employees Active users

7. October 2017
10 800+ 6M
Data Engineers Slack employees Active users

8. Data usage
1 in 3 per
week
500+
tables
400k
access data
warehouse
Tables Events per sec

9. It is all about Slogs

10. Well, not exactly

11. Slog

12. Slog

13. Druid infrastructure & usecases

14. What can go wrong?

15. We want more...

16. Performance & Experimentation
● Engineering & CE team should be able to detect the performance
bottleneck proactively.
● Engineers should be able to see their experimentation performance in
near real-time.

17. Near Real time Pipeline

18. Keep the load in DW Kafka predictable.
More comfortable to upgrade and verify newer Kafka version.
Smaller Kafka cluster is relatively more straightforward to operate.
Why Analytics Kafka

19. Druid Architecture

20. Middle manager Autoscale based on number of running tasks.
Historical node autoscale based on the segment size.
Fault tolerance deployment for overlord & Coordinator
Brokers autoscale and load balanced by ELB.
Druid Architecture

21. Challenges

22. Cascading failures

23. Forward Index fields

24. SQL

25. Bridge the gap between batch and
realtime tables.

26. Thank You!
26