This document discusses scalable data warehousing solutions on Hadoop. It introduces Hive LLAP for high performance SQL queries, Druid for dynamic OLAP cubes, and how they integrate with Hadoop for both real-time and batch analytics. The roadmap aims to provide a unified SQL and MDX layer for business intelligence tools on top of these technologies.

1. Scalable data warehousing
State of the art
Matthieu Lamairesse Hortonworks

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Scalable Data Warehousing on Hadoop
Overview
Solution Architecture
Demo
Roadmap

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
BI on Hadoop : What people think
COMPLEX
SLOW
SLOW
MR
BATCH
SQL
MDX
EXTRACT

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Reality : Differentiated data consumption
LLAP
Druid
DATA LAYER
Analyst
Spark
HiveACID
Data scientist
Admin & DBA
Relationnal
OLAP
Data Science
Business Intelligence

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
BI : A Tool for each use case
Slow but robust
à Optimized MR
à Batch Processing (SQL)
à ETL / ELT
à Static reports
LLAP
Interactive Ad Hoc / Exploratory Structured Interactive
Druid
à In Memory Cache
à Ad Hoc / Exploratory
à Interactive Dashboard
Þ No data transformation
Þ Interactive performance
Þ No Security setting changes
à OLAP cubes
à Fast dashboards
Tech
Query speed
Model
Use
cases
Relationnal
Full Joins / ACID
De-normalized
no Joins

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Scalable Data Warehousing on Hadoop
Overview
Solution Architecture
Demo
Roadmap

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks EDW Solution Components
Hadoop
Scalable Storage and Compute
Hive LLAP
High Performance SQL Data Mart
Druid
Dynamic OLAP Cubes for Higher
PerformanceFast, scalable SQL analytics
Intelligent in-memory caching
Define OLAP cubes for 10x faster queries
Unified semantic layer for all BI tools
Pre-aggregated
data
... Or, full-fidelity
data

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hive : At the heart of STRUCTURED data access on Hadoop
à SQL Standard ACID MERGE now
available in Hive.
à Efficiently perform record-level
inserts, updates and deletes.
à Delivers real Data Management in
Hadoop, massively simplifying
updates, data restatements and
change data capture.
ACID MERGE Makes Data Maintenance Simple
X Y
1 X1
5 X2
7 X3
A B
1 X1
2 Y2
3 Y3
4 Y4
5 X2
6 Y6
7 X3
Hadoop
Hive LLAP
Druid

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hive LLAP -- MPP Performance at Hadoop Scale
à Hybrid model combining daemons and containers for fast, concurrent execution of
analytical workloads (e.g. Hive SQL queries)
– Concurrent queries without specialized YARN queue setup
– Multi-threaded execution of vectorized operator pipelines
à Asynchronous IO and efficient in-memory caching
à How does it work ?
– Hive decides where query fragments
run (LLAP, Container, AM) based on
configuration, data size, format, etc
– Each Query coordinated
independently by a Tez AM
– Number of concurrent queries
throttled by number of active Ams
– Hive Operators used for processing
– Tez Runtime used for data transfer
Hadoop
Hive LLAP
Druid
Deep
Storag
e
YARN Cluster
LLAP Daemon
Query
Executors
LLAP Daemon
Query
Executors
LLAP Daemon
Query
Executors
LLAP Daemon
Query
Executors
Query
Coordinators
Coord-
inator
Coord-
inator
Coord-
inator
HiveServer2
(Query
Endpoint)
ODBC /
JDBC
SQL
Queries
In-Memory Cache
(Shared Across All Users)
HDFS and
Compatible
S3 WASB Isilon

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
HIVE : Continuously evolving SQL performance on Hadoop
= 100X
Hive + MR Hive + Tez Hive + Tez + LLAP
= 20X
Hive 1.0 Hive 1.3 Hive 2.0
Batch SQL
Analytics
SQL
Interactive
SQL
• ETL
• Reporting
• Data Mining
• Deep Analytics
• Reporting
• BI Tools:
Microstrategy,
Cognos
• Ad-Hoc
• Drill-Down
• Agile BI Tools:
Tableau, Power BI
Hadoop
Hive LLAP
Druid

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Druid High Level Architecture
Many nodes, based on
data size and #queries
HDFS or S3
Pivot or Dashboards,
BI Tools
Synthesize real-time and
historical information
Hadoop
Hive LLAP
Druid

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Columnar format + Dictionaries + Inverted Indexes = Speed
Hadoop
Hive LLAP
Druid

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hive / Druid Integration
à Build OLAP materialized views over Hive
data using Hive SQL.
à Merge historical and real-time data.
à Query using Hive and any tool that
supports Hive SQL.
Key Attributes:
OLAP Models
SQL Analytics
Streams and Archives (HDFS / Kafka)
Transformation Pipelines
CRMERP
Real-time
Events
Derived Cubes Derived Cubes
Facts and Historical
Events
(Hive LLAP)
Updatable
Dimensions
(Hive LLAP)
Full Fidelity Events
(Druid)
Aggregated Events
(Druid)
Dashboards

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Scalable Data Warehousing on Hadoop
Overview
Solution Architecture
Demo
Roadmap

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Démo – Trajets des taxis New Yorkais 2012 -2013
On y retrouve entres autres les indicateurs suivants :
à Timestamp de la course ( Début / Fin
à Coordonnées Géo (Début / Fin )
à Distance
à Type de paiement
à Couts de la course (Total, Pourboire, Taxes, … )
Le jeux de données décrit l’intégralité des trajets des taxi New Yorkais sur 2 ans
Infrastructure :
- 5 Data Nodes ( 8 vCPU, 15GB ram )
- 1 Maitre
- 1 Passerelle pour KNOX
Volumétrie :
- Table dé-normalisée de 351 646 964 entrées

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Scalable Data Warehousing on Hadoop
Overview
Solution Architecture
Demo
Roadmap

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Roadmap : End of Phase 1 – Current
UI
Core Platform
S3 or HDFS
HiveServer2
MDX
Unified SQL and MDX Layer
SQL BI Tools MDX Tools
Hive
Realtime Feeds
(Kafka, Storm, etc.)
Druid
OLAP Indexes
HiveServer2
Hive SQL
Thrift Server
SparkSQL
Fast SQL MDX
Superset UI
Fast Exploration
Builder UI
SmartSense
Ranger
Atlas
Ambari
Management

18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Roadmap : Target ( HDP 3.X )
UI
Core Platform
S3 or HDFS
HiveServer2
MDX
Unified SQL and MDX Layer
SQL BI Tools MDX Tools
Hive
Realtime Feeds
(Kafka, Storm, etc.)
Druid
OLAP Indexes
HiveServer2
Hive SQL
Thrift Server
SparkSQL
Fast SQL MDX
Superset UI
Fast Exploration
Builder UI
SmartSense
Ranger
Atlas
Ambari
Management

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hive + Druid = Insight When You Need It
OLAP Cubes SQL Tables
Streaming Data Historical Data
Unified SQL Layer
Pre-Aggregate ACID MERGE
Easily ingest event
data into OLAP cubes
Keep data up-to-date
with Hive MERGE
Build OLAP Cubes from Hive
Archive data to Hive for history
Run OLAP queries in real-time
or Deep Analytics over all history
Deep AnalyticsReal-Time Query
Materialized View
Navigation:
Transparent re-write of
queries to use the OLAP
index when possible
CALCITE-173

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved