Talks about what & why Druid ? Architecture Use cases in Real Time Analytics Query Patterns Industry adoption etc...

1. Interactive Analytics at Scale
By:-Suman Banerjee

2. Druid Concepts
• What is it ?
Druid is a open source fast distributed column-oriented data store.
Designed for low latency ingestion and very fast ad-hoc aggregation based
analytics.
• Pros:-
• Fast response in aggregation operation (in almost sub-second)
• Supports real time streaming ingestion capability with many other popular
solution in market e.g. Kafka , Samza , Spark etc.
• Traditional Batch type ingestion ( Hadoop based ).
• Cons / Limitation :-
• joins are not mature enough
• Limited options compared to other SQL like solutions.

3. Brief History on Druid
• History
• Druid was started in 2011 to power the analytics in Metamarkets. The project
was open-sourced to an Apache License in February 2015.

4. Industries has Druid in production
• Metamarkets
• Druid is the primary data store for Metamarkets’ full stack visual analytics service for the RTB (real time bidding) space. Ingesting over
30 billion events per day, Metamarkets is able to provide insight to its customers using complex ad-hoc queries at query time of
around 1 second in almost 95% of the time.
• Airbnb
• Druid powers slice and dice analytics on both historical and real time-time metrics. It significantly reduces latency of analytic queries
and help people to get insights more interactively.
• Alibaba
• At Alibaba Search Group, we use Druid for real-time analytics of users' interaction with its popular e-commerce site.
• Cisco
• Cisco uses Druid to power a real-time analytics platform for network flow data.
• eBay
• eBay uses Druid to aggregate multiple data streams for real-time user behavior analytics by ingesting up at a very high rate(over
100,000 events/sec), with the ability to query or aggregate data by any random combination of dimensions, and support over 100
concurrent queries without impacting ingest rate and query latencies.

5. Industries …

6. Druid In Production - MetaMarkets
• 3M+ events/sec through Druid’s real time ingestion.
• 100+ PB of data
• Application supporting 1000 of queries per sec concurrently.
• Supports 1000 of cores for horizontally scale up.
• …
• Reference :- https://metamarkets.com/2016/impact-on-query-speed-from-
forced-processing-ordering-in-druid/
• https://metamarkets.com/2016/distributing-data-in-druid-at-petabyte-
scale/

7. A real example of Druid in Action
Reference :- https://whynosql.com/2015/11/06/lambda-architecture-with-druid-at-gumgum/

8. Ideal requirements to Druid ?
• You need :-
• Fast aggregation & arbitrary data exploration in low latency on huge data sets.
• Fast response on near real time event data. Ingested data is immediately
available for querying)
• No SPoF
• Handle peta-bytes of data with multiple dimension.
• Less than a second in time-oriented summarization of the incoming data
stream
• NOTE >> before we go to understand the architecture part of it , I want to
show u a typical use case just to understand what we have said so far.

9. Druid Concepts – An example
• The Data
• timestamp publisher advertiser gender country click price
• 2011-01-01T01:01:35Z bieberfever.com google.com Male USA 0 0.65
• 2011-01-01T01:03:63Z bieberfever.com google.com Male USA 0 0.62
• 2011-01-01T01:04:51Z bieberfever.com google.com Male USA 1 0.45
• 2011-01-01T01:00:00Z ultratrimfast.com google.com Female UK 0 0.87
• 2011-01-01T02:00:00Z ultratrimfast.com google.com Female UK 0 0.99
• 2011-01-01T02:00:00Z ultratrimfast.com google.com Female UK 1 1.53
GROUP BY timestamp, publisher, advertiser, gender, country
:: impressions = COUNT(1), clicks = SUM(click), revenue = SUM(price)
timestamp publisher advertiser gender country impressions clicks revenue
2011-01-01T01:00:00Z ultratrimfast.com google.com Male USA 1800 25 15.70
2011-01-01T01:00:00Z bieberfever.com google.com Male USA 2912 42 29.18
2011-01-01T02:00:00Z ultratrimfast.com google.com Male UK 1953 17 17.31
2011-01-01T02:00:00Z bieberfever.com google.com Male UK 3194 170 34.01

10. Druid – Architecture
C
L
I
N
T
Indexing Streaming data
Real Time Node
Broker Node
Historical Node
H
D
F
S
Deep Storage
H
D
F
S
Over lord node
Indexing
DATA
QUERY
Static Data

11. H
D
F
S
Druid – Architecture ( cluster mgmt. depedency)
C
L
I
N
T
Indexing Streaming data
Real Time Node
Broker Node
Historical Node
Deep Storage
H
D
F
S
Coordinator node
DATA
QUERY
Zookeeper
Meta Store

12. Druid – Components
• Broker Node
• Real time node
• Overlord Node
• Middle-Manager Node
• Historical Node
• Coordinator Node
• Aside from these nodes, there are 3 external dependencies to the system:
• A running ZooKeeper cluster for cluster service discovery and maintenance of current
data topology
• A metadata storage instance for maintenance of metadata about the data segments that
should be served by the system
• A "deep storage" system to hold the stored segments.

13. Druid - Data Storage Layer
• Segments and Data Storage
• Druid stores its index in segment files, which are partitioned by time
• columnar: the data for each column is laid out in separate data structures.

14. Druid – Query
• Timeseries
• TopN
• GroupBy & Aggregations
• Time Boundary
• Search
• Select
• a) queryType
• b) granularity
• c) filter
• d) aggregation
• e) post-Aggregation

15. Demo

16. Task Submit Commands
• 1- clear HDFS storage location
• hdfs dfs –rm –r /user/root/segments
• Make sure the data source is exist in local FS :-
/root/labtest/druid_hadoop/druid-0.10.0/quickstart/Test/pageviewsLatforCountExmaple.json & upload to HDFS.
hdfs dfs -put -f pageviewsLat.json /user/root/quickstart/Test
• Create index Task on Druid
• curl -X 'POST' -H 'Content-Type:application/json' -d @quickstart/Test/pageviewsLat-index-forCountExample.json
localhost:8090/druid/indexer/v1/task
• Task information can be seen in <overlord_Host:8090>/console.html
• Verify the segments is created under /user/root/segments
• hdfs dfs –ls /user/root/segments

17. Query commands
• TopN
• This will result Top N pages with latency in descending order.
• curl -L -H'Content-Type: application/json' -XPOST --data-binary @quckstart/Test/query/pageviewsLatforCount-top-
latency-pages.json http://localhost:8082/druid/v2/?pretty
• Timeseries
• This will result total latency , filtered by user=“alice” and "granularity": "day“ . [ “all” ]
• curl -L -H'Content-Type: application/json' -XPOST --data-binary @ckstart/Test/query/pageviewsLatforCount-timeseries-
pages.json http://localhost:8082/druid/v2/?pretty
• groupBy
• A) This is will result aggregated latency grpBy user+url
• curl -L -H'Content-Type: application/json' -XPOST --data-binary
@quickstart/Test/query/pageviewsLatforCount-aggregateLatencyGrpByURLUser.json
http://localhost:8082/druid/v2/?pretty
• B) This will result aggregated page count (i.e. number of url accessed ) grpBy user
• curl -L -H'Content-Type: application/json' -XPOST --data-binary
@quickstart/Test/query/pageviewsLatforCount-countURLAccessedGrpByUser.json
http://localhost:8082/druid/v2/?pretty

18. Query commands
• Time Boundary
• Time boundary queries return the earliest and latest data points of a data set.
• curl -L -H'Content-Type: application/json' -XPOST --data-binary @ckstart/Test/query/pageviewsLatforCount-
timeBoundary-pages.json http://localhost:8082/druid/v2/?pretty
• Search
• A search query returns dimension values that match the search specification , like e.g. here searching for dimension
url has matches with text “facebook”
• curl -L -H'Content-Type: application/json' -XPOST --data-binary @ckstart/Test/query/pageviewsLatforCount-search-
URL-pages.json http://localhost:8082/druid/v2/?pretty

19. Thank You