Hive - Data Warehousing & Analytics on Hadoop Wednesday, June 10, 2009 Santa Clara Marriott Namit Jain, Zheng Shao Facebook

Agenda Introduction Facebook Usage Hive Progress and Roadmap Open Source Community Facebook

Introduction

Facebook Usage

Hive Progress and Roadmap

Open Source Community

Introduction Facebook

Introduction

Why Another Data Warehousing System? Data, data and more data ~1TB per day in March 2008 ~10TB per day today Facebook

Data, data and more data

~1TB per day in March 2008

~10TB per day today

Lets try Hadoop… Pros Superior in availability/scalability/manageability Efficiency not that great, but throw more hardware Partial Availability/resilience/scale more important than ACID Cons: Programmability and Metadata Map-reduce hard to program (users know sql/bash/python) Need to publish data in well known schemas Solution: HIVE Facebook

Pros

Superior in availability/scalability/manageability

Efficiency not that great, but throw more hardware

Partial Availability/resilience/scale more important than ACID

Cons: Programmability and Metadata

Map-reduce hard to program (users know sql/bash/python)

Need to publish data in well known schemas

Solution: HIVE

Lets try Hadoop… (continued) RDBMS> select key, count(1) from kv1 where key > 100 group by key; vs. $ cat > /tmp/reducer.sh uniq -c | awk '{print $2" "$1}‘ $ cat > /tmp/map.sh awk -F '01' '{if($1 > 100) print $1}‘ $ bin/hadoop jar contrib/hadoop-0.19.2-dev-streaming.jar -input /user/hive/warehouse/kv1 -mapper map.sh -file /tmp/reducer.sh -file /tmp/map.sh -reducer reducer.sh -output /tmp/largekey -numReduceTasks 1 $ bin/hadoop dfs –cat /tmp/largekey/part* Facebook

RDBMS> select key, count(1) from kv1 where key > 100 group by key;

vs.

$ cat > /tmp/reducer.sh

uniq -c | awk '{print $2" "$1}‘

$ cat > /tmp/map.sh

awk -F '01' '{if($1 > 100) print $1}‘

$ bin/hadoop jar contrib/hadoop-0.19.2-dev-streaming.jar -input /user/hive/warehouse/kv1 -mapper map.sh -file /tmp/reducer.sh -file /tmp/map.sh -reducer reducer.sh -output /tmp/largekey -numReduceTasks 1

$ bin/hadoop dfs –cat /tmp/largekey/part*

What is HIVE? A system for managing and querying structured data built on top of Hadoop Map-Reduce for execution HDFS for storage Metadata on raw files Key Building Principles: SQL as a familiar data warehousing tool Extensibility – Types, Functions, Formats, Scripts Scalability and Performance Facebook

A system for managing and querying structured data built on top of Hadoop

Map-Reduce for execution

HDFS for storage

Metadata on raw files

Key Building Principles:

SQL as a familiar data warehousing tool

Extensibility – Types, Functions, Formats, Scripts

Scalability and Performance

Simplifying Hadoop RDBMS> select key, count(1) from kv1 where key > 100 group by key; vs. hive> select key, count(1) from kv1 where key > 100 group by key; Facebook

RDBMS> select key, count(1) from kv1 where key > 100 group by key;

vs.

hive> select key, count(1) from kv1 where key > 100 group by key;

Facebook Usage Facebook

Facebook Usage

Data Warehousing at Facebook Today Facebook Web Servers Scribe Servers Filers Hive on Hadoop Cluster Oracle RAC Federated MySQL

Hive/Hadoop Usage @ Facebook Types of Applications: Reporting Eg: Daily/Weekly aggregations of impression/click counts SELECT pageid, count(1) as imps FROM imp_table GROUP BY pageid WHERE date = ‘2009-05-01’; Complex measures of user engagement Ad hoc Analysis Eg: how many group admins broken down by state/country Data Mining (Assembling training data) Eg: User Engagement as a function of user attributes Spam Detection Anomalous patterns for Site Integrity Application API usage patterns Ad Optimization Facebook

Types of Applications:

Reporting

Eg: Daily/Weekly aggregations of impression/click counts

SELECT pageid, count(1) as imps FROM imp_table GROUP BY pageid WHERE date = ‘2009-05-01’;

Complex measures of user engagement

Ad hoc Analysis

Eg: how many group admins broken down by state/country

Data Mining (Assembling training data)

Eg: User Engagement as a function of user attributes

Spam Detection

Anomalous patterns for Site Integrity

Application API usage patterns

Ad Optimization

Hadoop Usage @ Facebook Cluster Capacity: 600 nodes ~2.4PB (80% used) Data statistics: Source logs/day: 6TB Dimension data/day: 4TB Compression Factor ~5x (gzip) Usage statistics: 3200 jobs/day with 800K tasks(map-reduce tasks)/day 55TB of compressed data scanned daily 15TB of compressed output data written to hdfs 150 active users within Facebook Facebook

Cluster Capacity:

600 nodes

~2.4PB (80% used)

Data statistics:

Source logs/day: 6TB

Dimension data/day: 4TB

Compression Factor ~5x (gzip)

Usage statistics:

3200 jobs/day with 800K tasks(map-reduce tasks)/day

55TB of compressed data scanned daily

15TB of compressed output data written to hdfs

150 active users within Facebook

Hive Progress and Roadmap Facebook

Hive Progress and Roadmap

CREATE TABLE clicks(key STRING, value STRING) LOCATION '/hive/clicks' PARTITIONED BY (ds STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.TestSerDe' WITH SERDEPROPERTIES ('testserde.default.serialization.format'='03'); Facebook

CREATE TABLE clicks(key STRING, value STRING) LOCATION '/hive/clicks' PARTITIONED BY (ds STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.TestSerDe' WITH SERDEPROPERTIES ('testserde.default.serialization.format'='03');

Data Model Facebook Logical Partitioning Hash Partitioning clicks HDFS MetaStore /hive/clicks /hive/clicks/ds=2008-03-25 /hive/clicks/ds=2008-03-25/0 … Tables Metastore DB Data Location Bucketing Info Partitioning Cols

HIVE: Components Facebook HDFS Hive CLI DDL Queries Browsing Map Reduce MetaStore Thrift API SerDe Thrift CSV JSON.. Execution Parser Planner Web UI Optimizer DB

Hive Query Language SQL Subqueries in from clause Equi-joins Multi-table Insert Multi-group-by Sampling SELECT s.key, count(1) FROM clicks TABLESAMPLE (BUCKET 1 OUT OF 32) s WHERE s.ds = ‘2009-04-22’ GROUP BY s.key Facebook

SQL

Subqueries in from clause

Equi-joins

Multi-table Insert

Multi-group-by

Sampling

SELECT s.key, count(1) FROM clicks TABLESAMPLE (BUCKET 1 OUT OF 32) s WHERE s.ds = ‘2009-04-22’ GROUP BY s.key

FROM pv_users INSERT INTO TABLE pv_gender_sum SELECT gender, count(DISTINCT userid) GROUP BY gender INSERT INTO DIRECTORY ‘/user/facebook/tmp/pv_age_sum.dir’ SELECT age, count(DISTINCT userid) GROUP BY age INSERT INTO LOCAL DIRECTORY ‘/home/me/pv_age_sum.dir’ SELECT age, count(DISTINCT userid) GROUP BY age; Facebook

FROM pv_users

INSERT INTO TABLE pv_gender_sum

SELECT gender, count(DISTINCT userid)

GROUP BY gender

INSERT INTO DIRECTORY ‘/user/facebook/tmp/pv_age_sum.dir’

SELECT age, count(DISTINCT userid)

GROUP BY age

INSERT INTO LOCAL DIRECTORY ‘/home/me/pv_age_sum.dir’

SELECT age, count(DISTINCT userid)

GROUP BY age;

Hive Query Language (continued) Extensibility Pluggable Map-reduce scripts Pluggable User Defined Functions Pluggable User Defined Types Complex object types: List of Maps Pluggable Data Formats Apache Log Format Facebook

Extensibility

Pluggable Map-reduce scripts

Pluggable User Defined Functions

Pluggable User Defined Types

Complex object types: List of Maps

Pluggable Data Formats

Apache Log Format

FROM ( FROM pv_users MAP pv_users.userid, pv_users.date USING 'map_script‘ AS dt, uid CLUSTER BY dt) map INSERT INTO TABLE pv_users_reduced REDUCE map.dt, map.uid USING 'reduce_script' AS date, count; Pluggable Map-Reduce Scripts Facebook

FROM (

FROM pv_users

MAP pv_users.userid, pv_users.date

USING 'map_script‘

AS dt, uid

CLUSTER BY dt) map

INSERT INTO TABLE pv_users_reduced

REDUCE map.dt, map.uid

USING 'reduce_script'

AS date, count;

Map Reduce Example Facebook Machine 2 Machine 1 <k1, v1> <k2, v2> <k3, v3> <k4, v4> <k5, v5> <k6, v6> <nk1, nv1> <nk2, nv2> <nk3, nv3> <nk2, nv4> <nk2, nv5> <nk1, nv6> Local Map <nk2, nv4> <nk2, nv5> <nk2, nv2> <nk1, nv1> <nk3, nv3> <nk1, nv6> Global Shuffle <nk1, nv1> <nk1, nv6> <nk3, nv3> <nk2, nv4> <nk2, nv5> <nk2, nv2> Local Sort <nk2, 3> <nk1, 2> <nk3, 1> Local Reduce

Hive QL – Join INSERT INTO TABLE pv_users SELECT pv.pageid, u.age FROM page_view pv JOIN user u ON (pv.userid = u.userid); Facebook

INSERT INTO TABLE pv_users

SELECT pv.pageid, u.age

FROM page_view pv

JOIN user u

ON (pv.userid = u.userid);

Hive QL – Join in Map Reduce Facebook page_view user pv_users Map Reduce key value 111 < 1, 1> 111 < 1, 2> 222 < 1, 1> pageid userid time 1 111 9:08:01 2 111 9:08:13 1 222 9:08:14 userid age gender 111 25 female 222 32 male key value 111 < 2, 25> 222 < 2, 32> key value 111 < 1, 1> 111 < 1, 2> 111 < 2, 25> key value 222 < 1, 1> 222 < 2, 32> Shuffle Sort Pageid age 1 25 2 25 pageid age 1 32

Join Optimizations Map Joins User specified small tables stored in hash tables on the mapper backed by jdbm No reducer needed INSERT INTO TABLE pv_users SELECT /*+ MAPJOIN(pv) */ pv.pageid, u.age FROM page_view pv JOIN user u ON (pv.userid = u.userid); Future Exploit table/column statistics for deciding strategy Facebook

Map Joins

User specified small tables stored in hash tables on the mapper backed by jdbm

No reducer needed

INSERT INTO TABLE pv_users

SELECT /*+ MAPJOIN(pv) */ pv.pageid, u.age

FROM page_view pv JOIN user u

ON (pv.userid = u.userid);

Future

Exploit table/column statistics for deciding strategy

Hive QL – Map Join Facebook page_view user Hash table pv_users key value 111 <1,2> 222 <2> pageid userid time 1 111 9:08:01 2 111 9:08:13 1 222 9:08:14 userid age gender 111 25 female 222 32 male Pageid age 1 25 2 25 1 32

Hive QL – Group By SELECT pageid, age, count(1) FROM pv_users GROUP BY pageid, age; Facebook

SELECT pageid, age, count(1)

FROM pv_users

GROUP BY pageid, age;

Hive QL – Group By in Map Reduce Facebook pv_users Map Reduce pageid age 1 25 1 25 pageid age count 1 25 3 pageid age 2 32 1 25 key value <1,25> 2 key value <1,25> 1 <2,32> 1 key value <1,25> 2 <1,25> 1 key value <2,32> 1 Shuffle Sort pageid age count 2 32 1

Group by Optimizations Map side partial aggregations Hash-based aggregates Serialized key/values in hash tables 90% speed improvement on Query SELECT count(1) FROM t; Load balancing for data skew Optimizations being Worked On: Exploit pre-sorted data for distinct counts Exploit table/column statistics for deciding strategy Facebook

Map side partial aggregations

Hash-based aggregates

Serialized key/values in hash tables

90% speed improvement on Query

SELECT count(1) FROM t;

Load balancing for data skew

Optimizations being Worked On:

Exploit pre-sorted data for distinct counts

Exploit table/column statistics for deciding strategy

Columnar Storage CREATE table columnTable (key STRING, value STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.ColumnarSerDe' STORED AS RCFILE; Saved 25% of space compared with SequenceFile Based on one of the largest tables (30 columns) inside Facebook Both are compressed with GzipCodec Speed improvements in progress Need to propagate column-selection information to FileFormat *Contribution from Yongqiang He (outside Facebook) Facebook

CREATE table columnTable

(key STRING, value STRING)

ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.ColumnarSerDe'

STORED AS RCFILE;

Saved 25% of space compared with SequenceFile

Based on one of the largest tables (30 columns) inside Facebook

Both are compressed with GzipCodec

Speed improvements in progress

Need to propagate column-selection information to FileFormat

*Contribution from Yongqiang He (outside Facebook)

Speed Improvements over Time Facebook QueryA: SELECT count(1) FROM t; QueryB: SELECT concat(concast(concat(a,b),c),d) FROM t; QueryC: SELECT * FROM t; Time measured is map-side time only (to avoid unstable shuffling time at reducer side). It includes time for decompression and compression (both using GzipCodec). * No performance benchmarks for Map-side Join yet. Date SVN Revision Major Changes Query A Query B Query C 2/22/2009 746906 Before Lazy Deserialization 83 sec 98 sec 183 sec 2/23/2009 747293 Lazy Deserialization 40 sec 66 sec 185 sec 3/6/2009 751166 Map-side Aggregation 22 sec 67 sec 182 sec 4/29/2009 770074 Object Reuse 21 sec 49 sec 130 sec 6/3/2009 781633 Map-side Join * 21 sec 48 sec 132 sec

QueryA: SELECT count(1) FROM t;

QueryB: SELECT concat(concast(concat(a,b),c),d) FROM t;

QueryC: SELECT * FROM t;

Time measured is map-side time only (to avoid unstable shuffling time at reducer side). It includes time for decompression and compression (both using GzipCodec).

* No performance benchmarks for Map-side Join yet.

Overcoming Java Overhead Reuse objects Use Writable instead of Java Primitives Reuse objects across all rows *40% speed improvement on Query C Lazy deserialization Only deserialize the column when asked Very helpful for complex types (map/list/struct) *108% speed improvement on Query A Facebook

Reuse objects

Use Writable instead of Java Primitives

Reuse objects across all rows

*40% speed improvement on Query C

Lazy deserialization

Only deserialize the column when asked

Very helpful for complex types (map/list/struct)

*108% speed improvement on Query A

Generic UDF and UDAF Let UDF and UDAF accept complex-type parameters Integrate UDF and UDAF with Writables public IntWritable evaluate(IntWritable a, IntWritable b) { intWritable.set((int)(a.get() + b.get())); return intWritable; } Facebook

Let UDF and UDAF accept complex-type parameters

Integrate UDF and UDAF with Writables

public IntWritable evaluate(IntWritable a, IntWritable b) {

intWritable.set((int)(a.get() + b.get()));

return intWritable;

}

HQL Optimizations Predicate Pushdown Merging n-way join Column Pruning Facebook

Predicate Pushdown

Merging n-way join

Column Pruning

Open Source Community Facebook

Open Source Community

Open Source Community 21 contributors and growing 6 contributors within Facebook Contributors from: Academia Other web companies Etc.. 7 committers 1 external to Facebook and looking to add more here Facebook

21 contributors and growing

6 contributors within Facebook

Contributors from:

Academia

Other web companies

Etc..

7 committers

1 external to Facebook and looking to add more here

50 jiras fixed in last month 218 jiras still open 125 mails in last month on hive-user@ 600 mails in last month on hive-dev@ Various companies/universities Adknowledge, Admob Berkeley, Chinese Academy of Science Demonstration in VLDB’2009 Facebook

50 jiras fixed in last month

218 jiras still open

125 mails in last month on hive-user@

600 mails in last month on hive-dev@

Various companies/universities

Adknowledge, Admob

Berkeley, Chinese Academy of Science

Demonstration in VLDB’2009

Deployment Options EC2 http://wiki.apache.org/hadoop/Hive/HiveAws/HivingS3nRemotely Cloudera Virtual Machine http://www.cloudera.com/hadoop-training-hive-tutorial Your own cluster http://wiki.apache.org/hadoop/Hive/GettingStarted Hive can directly consume data on hadoop CREATE EXTERNAL TABLE mytable (key STRING, value STRING) LOCATION '/user/abc/mytable'; Facebook

EC2

http://wiki.apache.org/hadoop/Hive/HiveAws/HivingS3nRemotely

Cloudera Virtual Machine

http://www.cloudera.com/hadoop-training-hive-tutorial

Your own cluster

http://wiki.apache.org/hadoop/Hive/GettingStarted

Hive can directly consume data on hadoop

CREATE EXTERNAL TABLE mytable (key STRING, value STRING) LOCATION '/user/abc/mytable';

Future Work Benchmark & Performance Integration with BI tools (through JDBC/ODBC) Indexing More on Hive Roadmap http://wiki.apache.org/hadoop/Hive/Roadmap Machine Learning Integration Real-time Streaming Facebook

Benchmark & Performance

Integration with BI tools (through JDBC/ODBC)

Indexing

More on Hive Roadmap

http://wiki.apache.org/hadoop/Hive/Roadmap

Machine Learning Integration

Real-time Streaming

Information Available as a sub project in Hadoop http://wiki.apache.org/hadoop/Hive (wiki) http://hadoop.apache.org/hive (home page) http://svn.apache.org/repos/asf/hadoop/hive (SVN repo) ##hive (IRC) Works with hadoop-0.17, 0.18, 0.19 Release 0.3 is out and more are coming Mailing Lists: hive-{user,dev,commits}@hadoop.apache.org Facebook

Available as a sub project in Hadoop

http://wiki.apache.org/hadoop/Hive (wiki)

http://hadoop.apache.org/hive (home page)

http://svn.apache.org/repos/asf/hadoop/hive (SVN repo)

##hive (IRC)

Works with hadoop-0.17, 0.18, 0.19

Release 0.3 is out and more are coming

Mailing Lists:

hive-{user,dev,commits}@hadoop.apache.org

Contributors Aaron Newton Ashish Thusoo David Phillips Dhruba Borthakur Edward Capriolo Eric Hwang Hao Liu He Yongqiang Jeff Hammerbacher Johan Oskarsson Josh Ferguson Joydeep Sen Sarma Kim P. Facebook Michi Mutsuzaki Min Zhou Namit Jain Neil Conway Pete Wyckoff Prasad Chakka Raghotham Murthy Richard Lee Shyam Sundar Sarkar Suresh Antony Venky Iyer Zheng Shao

Aaron Newton

Ashish Thusoo

David Phillips

Dhruba Borthakur

Edward Capriolo

Eric Hwang

Hao Liu

He Yongqiang

Jeff Hammerbacher

Johan Oskarsson

Josh Ferguson

Joydeep Sen Sarma

Kim P.

Michi Mutsuzaki

Min Zhou

Namit Jain

Neil Conway

Pete Wyckoff

Prasad Chakka

Raghotham Murthy

Richard Lee

Shyam Sundar Sarkar

Suresh Antony

Venky Iyer

Zheng Shao

Questions Facebook

Questions