Apache Druid is a high-performance distributed analytics store for modern analytics applications. It supports ingesting millions of events per second and sub-second query processing. Druid supports various types of data sources for ingestion, including Apache Kafka. You can immediately query on stream events once they get ingested into Druid. Since Kafka provides scalable and robust data delivery while Druid supports advanced complex analysis on streams, Kafka and Druid are widely used together for BI and operational analytics use cases, which require interactivity, scalability, real-time, and performance. This talk is based on our real-world experiences building out streaming analytics stacks powering production use cases across many industries.

1. A complete data platform for
real-time analytics
Apr 2021
Druid + Kafka
Transform your data-in-motion to analytics-in-motion
July 2021
© 2021 Imply Data, Inc.

2. © 2021 Imply Data, Inc.
Who am I?
Gian Merlino
Committer & PMC chair at
Cofounder at (we’re hiring!)

3. © 2021 Imply Data, Inc.
Agenda
What does it mean to set analytics in motion?
Introduction to Apache Druid
Learn how Druid + Kafka work together
Try this at home

4. © 2021 Imply Data, Inc.
What does it mean to set analytics in motion?

5. © 2021 Imply Data, Inc.
Analytics-in-motion requirements
Interactive queries
Sub-second OLAP queries
Highly scalable
Massive data volumes, query
concurrency, ingestion throughput
Real-time
Ingest in true real-time from Kafka
Cost-effective
Fastest queries at lowest cost

6. © 2021 Imply Data, Inc.
Analytics-in-motion applications

7. © 2021 Imply Data, Inc.
Intro to Apache Druid

8. © 2021 Imply Data, Inc.
Druid is a powerful real-time database
=

9. © 2021 Imply Data, Inc.
Druid is a fancy calculator
=

10. © 2021 Imply Data, Inc.
Druid in the wild

11. © 2021 Imply Data, Inc.
Some numbers from Imply customers
Petabytes of data under management
1000s of daily active users
Millions of rows/second ingested
90%ile query time < 1s
99%ile query time < 2s

12. © 2021 Imply Data, Inc.
Druid is designed for real-time performance
Data sourced from: Correia, José & Costa, Carlos & Santos, Maribel. (2019). Challenging SQL-on-Hadoop Performance with Apache Druid.

13. © 2021 Imply Data, Inc.
Druid is designed for real-time performance
Data layer
Deep storage
Query layer
Streaming data is
immediately queryable
Hot, already-ingested
data cached locally
Filtered and aggregated data
transferred to query layer
Ingested data
stored in indexed
columnar format

14. © 2021 Imply Data, Inc.
How Druid + Kafka work together

15. © 2021 Imply Data, Inc.
Stream-only pattern
Kafka
Data
sources
Real-time
processing
Real-time
delivery
Kafka Streams
ksqlDB
Real-time
database
Presentation

16. © 2021 Imply Data, Inc.
Kafka + data lake pattern
Data lake
Kafka
Data
sources
File-based
storage
Data lake
Data lake
Query
engine
Query
engine
Query
engine
🔥 Analytics in motion
Federated query engine
(Presto, Hive, Spark, …)
Real-time
delivery
Direct load from Kafka topic
Backfill if
needed

17. © 2021 Imply Data, Inc.
Demo!

18. © 2021 Imply Data, Inc.
Try this at home

19. © 2021 Imply Data, Inc.
Stay in touch
19
@druidio
Join the community!
https://druid.apache.org/community/
Free online training by Imply!
https://imply.io/druid-university
Follow the Druid project on Twitter!

20. © 2021 Imply Data, Inc.
Run with Imply
20
Available at https://imply.io/
Point and click deployment
AWS, GCP, Kubernetes
Bring-your-own VM / bare metal
Batteries included: visualizations,
dashboarding, alerts

21. A complete data platform for
real-time analytics
Apr 2021
Thank you for attending
July 2021
© 2021 Imply Data, Inc.