Deep dive into Druid, from DruidIL Meetup @ Singular

1. Deep Dive Into Druid
Sep 2017

2. Agenda
➔ Where is druid??
➔ Columnar Databases / Distributed Database
➔ Druid Concepts
➔ Indexing Process
➔ Querying Process
➔ Some Benchmarks
➔ Monitoring / Logging
2

3. 3
Columnar Databases
+
Distributed Databases

4. Row Store
4

5. Row Store - On Disk
5

6. Column Store
6

7. Column Store
7

8. Column Store - On Disk
8

9. Timestamped Column Store
9

10. Distributed Database - Sharding
10

11. Distributed Database - Sharding
11

12. 12
Druid Concepts

13. Druid Concepts
What is Druid?
• OLAP
• Columnar
• Real-Time
• Distributed
• Optimized for Aggregations
• Optimized for query performance
• Horizontally Scale
13

14. Druid Concepts
What it’s not :
• Relational - (no joins!)
– lookup feature as partial solution
• Good at ad-hoc row retrieval
• Fully ACID
– no transactions
– eventually consisted
• Easy to change/delete data
• Simple
14

15. Data Model
• Cluster - Instance of Druid system
• Data Source - collection of data, equivalent for Table
• Segment - Immutable Timestamped data part of data-source
(file)
• Shard / Partition - For big segments, enabling partitions split the
segment
15
Data Source
“Tracker_Stats_Zynga”
Segment
May-17
Segment
June-17
Partition
P0
Partition
P1
Partition
P0
Partition
P1

16. Druid Concepts
16
S3 /
HDFS
Segments Store
Deep Storage
Batch / RT
Overlord
MiddleManagers
Indexing Service
Brokers
Historicals
Query Layer
Coordinator
ZooKeeper
Metadata-DB
Management

17. 17
Indexing Service

18. Indexing Process - Batch Ingestion Task
18
Overlord
Singular
Load-Launcher
API
Pending Waiting Running Completed
PeonPeon PeonPeon
MiddleManager MiddleManager
EMR Indexing
To S3
Singular
Load-Monitor
ZooKeeper
Metadata-DB

19. Indexing Console
19

20. Indexing Console
20

21. 21
Querying Infra

22. Metadata-DB
Loading Segments For Query
22
Deep Storage
Coordinator
Historicals
Historicals
Historicals
Segments
Disk
RAM
Segments

23. Coordinator Console
23

24. Query Process
24
Singular
WebApp / API
Historicals
Historicals
Historicals
Broker
Broker
Aggregations
Lookups
Segments
Disk
RAM
Cache

25. 25
Numbers

26. Numbers
➔ Druid Servers (Prod): 20
➔ EMR Indexing Servers: 32
➔ AVG Indexing Task Time: 4 Minutes
➔ Daily Indexing Tasks: 15,000
➔ Total Cluster Size: 3 TB
➔ Time to deploy a new cluster: 5 Minutes
26

27. The End
Questions?
27