Hive provides a mechanism for querying and managing structured data within Hadoop. It allows users familiar with SQL to query large datasets without needing to write MapReduce code. Hive uses HDFS for storage and MapReduce for execution, and supports SQL-like queries, aggregation, joins, and user-defined functions. It is designed to handle large datasets beyond the capabilities of traditional systems.

1. Hive: Data Warehousing for
Hadoop
Ben Lever
@bmlever
Big Data Analytics Meetup
27 March 2012

2. Another Data Warehousing System?
• Problem:
– Lots of data
• Partial solution:
– Hadoop
• Another problem:
– MapReduce can be hard
– Schema information embedded in program – a lot
of data is still structured

3. Solution: Hive
• A system for querying and managing
structured data within Hadoop
– MapReduce for execution
– HDFS for storage
• Designed for end-users that know more SQL
than Java
• Apache v2
• hive.apache.org

4. Working example: MovieLens
• Movie ratings
• 3 “tables”:
Users Movies Ratings
id id user id
age title movie id
gender release date rating (1 – 5)
occupation action timestamp
zip code adventure
romance
...
www.grouplens.org

5. Demo

6. So far
• Hive shell
• Creating and loading tables
• Data model:
– INT, BIGINT, TINYINT, STRING, etc
– Also: FLOAT, DOUBLE, ARRAY, MAP, STRUCT
• Simple queries with filtering
• Table data is immutable
• Schema on readvsschema on write

7. Hive components
TABLE customer (
customer_id BIGINT,
Metastore gender STRING,
...
schema info
launch MapReduce
Driver MapReduc
e job
Hive query
HDFS
(SQL-like)
raw source data
(compressed)
SELECT *
FROM customers CLI
WHERE gender = ‘M’;

8. Metastore
Hadoop – The Definitive Guide

9. Other SQL-like features
• Aggregation – COUNT, AVG
• JOIN
• GROUP BY
• SORT BY
• Sub queries

10. Demo

11. Built in functions
• Text mining:
– ngrams()
– context_ngrams()
– sentences()
• Statistics + mathematics:
– stddev()
– histogram_numeric()
– log
– radians

12. User Defined Functions
• Written in Java
• User Defined Functions (UDFs):
– Single row  Single row
– e.g. mathematical and string functions
• User Defined Aggregate Functions (UDAFs):
– Multiple rows  Single row
– e.g. AVG
• User Defined Table Functions (UDTFs):
– Single row  Multiple rows
– e.g. “explode”

13. Hive Clients
Hadoop – The Definitive Guide

14. Hive Server
JDBC ODBC

15. Sqoop
Move data between Hadoop
and relational databases
RDBMS Sqoop Hadoop
Hive
Metastore
schema
http://incubator.apache.org/projects/sqoop.html

16. Sqoop adapters

17. Conclusion
• Scales to handle much more data than traditional
systems:
– Leverages Hadoop HDFS and MapReduce
– Relational/structured data
– Schema on read vs schema on write
• Supports rapid iteration of ad-hoc queries
– SQL-like querying language
– Complex queries (joins, etc) with minimal code
• Is not a database replacement:
– Treats data as immutable
– No indexing