This document summarizes the key components and collaborations in Apache Druid. It describes Zookeeper's role in coordination, the Overlord's role in task management, the Broker's role in query routing, and the Middle Manager's role in ingestion and indexing. It provides diagrams illustrating how these components work together to ingest and store distributed data, and answer queries in a scalable way.

1. Apache®, Apache Druid®, Druid®, and the Druid logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.

2. peter.marshall@imply.io
20 years in Enterprise Architecture
CRM, EDRM, ERP, EIP, Digital Services,
Security, BI, RI, and MDM
BA Theology (!) and Computer Studies
TOGAF certified
Book collector & A/V buyer
Prime Timeline = proper timeline
#werk
petermarshall.io

3. What the collaborations do
Some principles
Each collaboration
Wrap-up
Questions!

4. Query
Distributed execution of SQL / Druid
Native queries on the cluster
Ingestion
Ingestion tasks that bring data into Druid
from storage and delivery services
Distribution
Replication and distribution of the
ingested data according to rules

6. ● A job to do
● Compute to do the ingestion
● A place to store optimised data
● A question to answer
● Data to process!
● Compute to answer queries
● Somewhere to put the data that’s
near to the query process
● Some rules to follow
Query
Distributed execution of SQL / Druid
Native queries on the cluster
Ingestion
Ingestion tasks that bring data into Druid
from storage and delivery services
Distribution
Replication and distribution of the
ingested data according to rules

7. https://www.archimatetool.com/

8. What the collaborations do
Some principles
Each collaboration
Wrap-up
Questions!

9. Zookeeper
Overlord
Ingest
Spec
Task
Log
S3, GCS, ABS, HFDS, HTTP, Local, MySql, Postgres, Druid
Kafka Kinesis
Middle Manager
(Indexer)

10. Zookeeper
Overlord
Middle Manager
(Indexer)

11. Deep Store
Zookeeper
Overlord
Middle Manager
(Indexer) columnarise
index & encode
time-shardDeep Store

12. Deep Store
Metadata
Store
Zookeeper
Overlord
Middle Manager
(Indexer)

13. Metadata
Store
Broker
Middle Manager
(Indexer)
Overlord
Deep Store
Zookeeper
SQL
and
Native

14. Metadata
Store
Zookeeper
Broker
Middle Manager
(Indexer)
Overlord
Deep Store

15. Metadata
Store
Zookeeper
Broker
Middle Manager
(Indexer)
Overlord
Deep Store Historical

16. Middle Manager
(Indexer)
Metadata
Store
Zookeeper
Historical
BrokerOverlord
Deep Store

17. OverlordOverlordCoordinator
Metadata
Store
Historical
Zookeeper
BrokerOverlord
Middle Manager
(Indexer)
Deep Store

18. OverlordOverlordCoordinator
Metadata
Store
Historical
Zookeeper
BrokerOverlord
Middle Manager
(Indexer)
Deep Store

19. Zookeeper
Historical
OverlordOverlordCoordinator
Metadata
Store
Historical
BrokerOverlord
Middle Manager
(Indexer)
Deep Store
Historical

20. Zookeeper
Historical
Historical
OverlordOverlordCoordinator
Metadata
Store
Historical
BrokerOverlord
Middle Manager
(Indexer)
Deep Store

23. What the collaborations do
Some principles
Each collaboration
Wrap-up
Questions!

28. https://www.archimatetool.com/

29. Zookeeper
Coordinator
Overlord
Broker
Query
Distributed execution of SQL / Druid
Native queries on the cluster
Ingestion
Ingestion tasks that bring data into Druid
from storage and delivery services
Distribution
Replication and distribution of the
ingested data according to rules

30. ★ A job to do
★ Compute to do the ingestion
★ A place to store optimised data
★ A question to answer
★ Data to process!
★ Compute to answer queries
★ Somewhere to put the data that’s
near to the query process
★ Some rules to follow
Query
Distributed execution of SQL / Druid
Native queries on the cluster
Ingestion
Ingestion tasks that bring data into Druid
from storage and delivery services
Distribution
Replication and distribution of the
ingested data according to rules

31. http://druid.apache.org
Imply Distribution
https://imply.io/get-started
@druidio
Add Apache Druid
as a skill
Apache Distribution
https://github.com/apache/druid
ASF Slack
#druid
Druid Community
https://druid.apache.org/community/
Meetup Groups
https://www.meetup.com/pro/apache-druid/
Google Groups Druid User Forum
https://groups.google.com/

32. What the collaborations do
Some principles
Each collaboration
Wrap-up
Questions!