This document discusses how Hadoop can be used in data warehousing and analytics. It begins with an overview of data warehousing and analytical databases. It then describes how organizations traditionally separate transactional and analytical systems and use extract, transform, load processes to move data between them. The document proposes using Hadoop as an alternative to traditional data warehousing architectures by using it for extraction, transformation, loading, and even serving analytical queries.

1. Hadoop and the Data Warehouse
Patrick Angeles
1

2. About Me
• Director of Field Engineering at Cloudera
• Architect on several dozen Hadoop-based data solutions
for Cloudera customers
• Started with Hadoop in 2008
• First Hadoop system processed set-top box log data
• Past life
• Java EE / Database Architect
• Web Data Mining
• Cryptography / Public Key Infrastructure
2

3. What is a Data Warehouse?
3

4. — The Oracle
4

5. Database Architecture 1.0
Products
Inventory
Customers DB
Sales
Orders
5

6. Database Architecture 1.0
• Dead simple
• Tables in 3rd normal form
• Reports are SQL queries that join through entity
relationships and aggregate
SELECT c.gender, p.product_name,
sum(o.qty), sum(o.price)
FROM order o, customer c, product p
WHERE o.customer_id = c.id
AND o.product_id = p.id
AND o.day = ’2013-03-21’
GROUP BY c.gender, p.product_name ;
6

7. Database Architecture 1.0
• Report queries can become expensive, redundant
• Build a layer of abstraction!
• Materialize the data to something closer to query
form.
• Create reporting tables
• Decide on the reports columns
• What query criteria can be parameterized
• Periodicity of report generation
• Denormalize and aggregate
7

8. Database Architecture 1.1
Inventory
Customers
Sales
Orders
Products
8

9. Two Database Workloads
Transactional Analytic
Record facts Reveal patterns
Write-optimized Read-optimized
Random reads/writes Sequential reads
Normalized schema Denormalized schema
9

10. Analytical Database (2.0)
Customers Inventory
Orders Sales
Products
10

11. Analytical Database Architecture
• Column oriented storage
• Reduces I/O on multi-dimensional tables
• Improved compression
• Skip columns or row ranges
• Massively Parallel Processing
• Query planner breaks up a task to be executed on
multiple hosts
• Shared-nothing Architecture
• Cluster nodes have independent storage and memory
• Slow writes, fast reads
11

12. Analytical Database
TX Analytical
DB DB
12

13. Data Transformation
TX Analytical
DB DB
13

14. Three Ways to Transform Data
• Transform Extract Load
• Query from transactional tables into target schema
• Extract Load Transform
• Load data into analytical database, transform and write
to target schema
• No need for additional hardware
• Extract Transform Load
• Read data from transactional database into a grid
system, transform, then write to analytical database
• Least load on tx and analytical systems
14

15. Business Intelligence Tools
TX Analytical
BI
DB DB
15

16. Business Intelligence Tools
• Can provide canned reports, dashboards, or
interactive visualizations
• Typically leverage common standards (SQL,
JDBC/ODBC) to access data
• Requires low-latency (sub second or minute,
depending on query) response times from database
16

17. Observations
• Separate transactional from analytical workloads
• Use appropriate database implementation
according to the workload
• ‘Traditional’ row-major store for transactional
• MPP column-store for analytic
• Consider a BI tool so you’re not stuck writing
reports for analysts who don’t know SQL
• Consider an ETL tool so you’re not stuck writing
transformations for analysts who don’t know SQL
17

18. Welcome to the Enterprise
18

19. Basic Data Warehouse Architecture
TX BI
DW
DB
19

20. Data Marts
Sales
TX Mktg BI
DW
DB
Prch
20

21. Multiple Data Sources
TX
DB Sales
Files DW Mktg BI
other Prch
21

22. Operational Data Store
TX
DB Sales
Files Mktg BI
ODS DW
other Prch
22

23. Where’s Hadoop?
23

24. No Hadoop
TX
DB Sales
Files Mktg BI
ODS DW
other Prch
24

25. Adjacent System
TX
DB Sales
Files Mktg BI
DW
ODS
other Prch
25

26. ETL Engine
TX
DB Sales
Files Mktg BI
DW
other Prch
26

27. Tiered Data Warehouse
TX
DB Sales
Files Mktg BI
other Prch
27

28. Analytical Query Engine
TX
DB
Files BI
other
28

29. Simple Database Architecture
Products
Inventory
Customers DB Sales
Orders
29

30. The future?
Products
Inventory
Customers
Sales
Orders
30

31. http://www.hbasecon.com/
San Francisco
June 13, 2013
31

32. 32