Dremel is a system that can run aggregate queries over trillions of read-only data in seconds. It uses a nested columnar data storage model and a multi-level serving tree to distribute queries across a cluster. Experiments showed that Dremel was able to process queries orders of magnitude faster than traditional MapReduce approaches on large datasets, and could scale to handle increasing volumes of data and query loads. Dremel's success led Google to develop BigQuery for interactive analysis of petabytes of data, and influenced the open source Apache Drill project.

1. Dremel: Interactive Analysis
of Web-Scale Datasets
Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar,
Matt Tolton, Theo Vassilakis
1
Presented by : Diwas Pandey

2. WASHINGTON
STATE
UNIVERSITY
2
DATA
[ Image Source: ResearchGate - Rainer Lenz ]

3. WASHINGTON
STATE
UNIVERSITY
3
DATA DATA ANALYSIS REPORTS
WHAT WE DO WITH DATA ??

4. WASHINGTON
STATE
UNIVERSITY
EXISTING TOOLS !!
4
❖ Hive
➢ allows users to read, write, and manage petabytes of
data using SQL
❖ Pig
➢ tool which is used to process the large datasets
❖ MapReduce
➢ allows data analysis for massive datasets using a
SQL-style syntax
➢ works on record oriented data

5. WASHINGTON
STATE
UNIVERSITY
SPEED MATTERS !!
5

6. WASHINGTON
STATE
UNIVERSITY
6
DREMEL

7. WASHINGTON
STATE
UNIVERSITY
DREMEL
7
❖ System that can run aggregate queries over trillions of
read-only data in seconds
❖ Scalable, Fault Tolerant & Fast
❖ Supports Nested data model
➢ Columnar storage of nested (non-relational) data
➢ Tree like architecture similar to web search
❖ Access data in situ (Eg. – GFS, Bigtable)
❖ Often used in conjunction with MapReduce
❖ Used by Google since 2006
Dremel : Speed

8. WASHINGTON
STATE
UNIVERSITY
RECORDS VS COLUMNS
8
❖ Advantages: Read Less, Fast Access

9. WASHINGTON
STATE
UNIVERSITY
NESTED COLUMNAR STORAGE
9

10. WASHINGTON
STATE
UNIVERSITY
10
R & D LEVEL
1 2 3
❖ r = 0 if first record
❖ d = branch

11. WASHINGTON
STATE
UNIVERSITY
11
R & D LEVEL
R & D LEVEL
1 2 3
❖ r = total sub branch;
❖ d = branch

12. WASHINGTON
STATE
UNIVERSITY
12
R & D LEVEL
1 2 3
❖ r = total sub branch;
❖ d = branch

13. WASHINGTON
STATE
UNIVERSITY
13
R & D LEVEL
1 2 3
❖ r = total sub branch;
❖ d = branch

14. WASHINGTON
STATE
UNIVERSITY
14
R & D LEVEL
❖ r = 0 if first record
❖ d = branch

15. WASHINGTON
STATE
UNIVERSITY
NESTED COLUMNAR STORAGE
15

16. WASHINGTON
STATE
UNIVERSITY
RECORD ASSEMBLY ALGORITHM
16
❖ Splitting Records into Columns

17. WASHINGTON
STATE
UNIVERSITY
QUERY LANGUAGE EXAMPLE
17

18. WASHINGTON
STATE
UNIVERSITY
QUERY EXECUTION
18
❖ Dremel uses a multi-level serving tree to
execute queries
❖ A root server receives incoming queries,
and routes the queries to the next level in
the serving tree.
❖ The leaf servers communicate with the
storage layer or access the data on local
disk.
Fig: System architecture and execution inside a server node

19. WASHINGTON
STATE
UNIVERSITY
QUERY DISPATCHER
19
❖ In Dremel, several queries are executed simultaneously
❖ Tablets, horizontal partitions of the table, are usually three-way replicated
❖ Fault tolerant; if one replica fails or is unavailable; reschedules on another
server

20. WASHINGTON
STATE
UNIVERSITY
20
EXPERIMENTS

21. WASHINGTON
STATE
UNIVERSITY
DATASET USED
21

22. WASHINGTON
STATE
UNIVERSITY
READ FROM DISK
22

23. WASHINGTON
STATE
UNIVERSITY
MR vs Dremel Execution
23

24. WASHINGTON
STATE
UNIVERSITY
SERVING TREE DEPTH
24
Intermediate Server :- 1:2900, 1:10:2900, 1:10:100:2900

25. WASHINGTON
STATE
UNIVERSITY
SCALABILITY
25

26. WASHINGTON
STATE
UNIVERSITY
INTERACTIVE SPEED
26
❖ 99%+ completed in 5 seconds

27. WASHINGTON
STATE
UNIVERSITY
27
WHAT HAPPENED NEXT ?

28. WASHINGTON
STATE
UNIVERSITY
DREMEL LED TO !!
28
❖ Google Big-Query
❖ Google Big-Query
❖ Apache Drill
➢ Open source Implementation of
BigQuery
Fig: System architecture of Big Query

29. WASHINGTON
STATE
UNIVERSITY
Questions?
29