Hive is an open-source data warehouse software project that facilitates querying and managing large datasets residing in distributed storage. It provides SQL-like queries to analyze large datasets using Hadoop. Key features include support for HDFS, HBase, and SQL-like language called HiveQL. Hive also supports user-defined functions, aggregation functions, and table-generating functions to extend its capabilities.

1. HiveHive 介绍介绍
周海汉周海汉 2013.4.182013.4.18

2. 目录目录
• HiveHive 介简介简
• HiveHive 特性特性
• HiveQLHiveQL
• UDFUDF
• 小技巧小技巧
• 讨论讨论

3. HiveHive 介简介简
• 官网官网
http://hive.apache.org/http://hive.apache.org/
• 最新版本最新版本 0.100.10
• facebookfacebook 献贡 给献贡 给 apacheapache

4. HiveHive 模式模式
• Metadb : embedded Derby database,mysql,otherMetadb : embedded Derby database,mysql,other
• local mode: Derbylocal mode: Derby ，， one userone user ，， one jobone job
• distribute mode: mysqldistribute mode: mysql ，， multi usermulti user

5. 支持支持 HadoopHadoop 版本版本
• hadoop 0.20~hadoop 0.20~
• hadoop 0.23~hadoop 0.23~

6. HiveHive 特性特性

7. HiveHive 特性特性
• 数据仓库数据仓库
• HiveQLHiveQL
• HDFS & HBaseHDFS & HBase
• ^A^A 分隔的行分隔的行

8. HiveHive 特性特性

9. HiveQLHiveQL

10. HiveQL - SQLHiveQL - SQL 部分子集部分子集
• No Update or Delete statements.No Update or Delete statements.
• each query tables only from one databaseeach query tables only from one database
• not support IN/EXISTS, Having clausenot support IN/EXISTS, Having clause
• ......

11. HiveQL -HiveQL - 超出超出 SQLSQL 部分部分
• 数据复杂 结构数据复杂 结构
• structstruct
• arrayarray
• mapmap
• ......

12. HiveQL -HiveQL - 自 的部分函数带自 的部分函数带
• ：统计：统计
– sum,count,avg,min,maxsum,count,avg,min,max
– 体 准差函数总 标体 准差函数总 标 : stddev_pop: stddev_pop
– 本 准差函数样 标本 准差函数样 标 : stddev_samp: stddev_samp
– 中位数函数中位数函数 : percentile: percentile
– 直方图直方图 : histogram_numeric: histogram_numeric
• 条件：条件：
– ifif
– casecase

13. HiveQL -HiveQL - 自 的部分函数带自 的部分函数带
• 时间时间 year date unix_timestamp ...year date unix_timestamp ...
• 逻辑逻辑 and or notand or not
• 算符运算符运 +-*/ % | & ^ ~+-*/ % | & ^ ~
• 数学数学 round floor ceil rand exp log log2 pow sqrt hexround floor ceil rand exp log log2 pow sqrt hex
sin ...sin ...
• 字符串 理处字符串 理处 trim substr length split get_json_objecttrim substr length split get_json_object
parse_url regexp_replace regexp_extractparse_url regexp_replace regexp_extract

14. HiveQLHiveQL 示例示例 -- 建创建创 HDFSHDFS 文本外表文本外表
• CREATE EXTERNAL TABLE login(CREATE EXTERNAL TABLE login(
• ldate string,ldate string,
• userid int,userid int,
• proid int,proid int,
• imei string,imei string,
• sysver string)sysver string)
• ROW FORMAT DELIMITED FIELDSROW FORMAT DELIMITED FIELDS
TERMINATED BY ' 'TERMINATED BY ' '
• LOCATIONLOCATION
• 'hdfs://h46:9000/flume/loginlog''hdfs://h46:9000/flume/loginlog'

15. HiveQLHiveQL 示例示例 -HBase-HBase 外表外表
• CREATE EXTERNAL TABLE lordstat_pid(CREATE EXTERNAL TABLE lordstat_pid(
• key string COMMENT 'from deserializer',key string COMMENT 'from deserializer',
• total int COMMENT 'from deserializer',total int COMMENT 'from deserializer',
• win int COMMENT 'from deserializer',win int COMMENT 'from deserializer',
• spring int COMMENT 'from deserializer')spring int COMMENT 'from deserializer')
• ROW FORMAT SERDEROW FORMAT SERDE
• 'org.apache.hadoop.hive.hbase.HBaseSerDe''org.apache.hadoop.hive.hbase.HBaseSerDe'
• STORED BYSTORED BY
• 'org.apache.hadoop.hive.hbase.HBaseStorageHandler''org.apache.hadoop.hive.hbase.HBaseStorageHandler'
• WITH SERDEPROPERTIES (WITH SERDEPROPERTIES (
• 'serialization.format'='1','serialization.format'='1',
• 'hbase.columns.mapping'=':key,i:t,i:win,i:spr')'hbase.columns.mapping'=':key,i:t,i:win,i:spr')
• TBLPROPERTIES (TBLPROPERTIES (
• 'hbase.table.name'='lordstat_pid');'hbase.table.name'='lordstat_pid');

16. HiveQLHiveQL 示例示例 -- 分区分区
• hive> create external table glog1(ldate string,ltime string ,threadidhive> create external table glog1(ldate string,ltime string ,threadid
string,userid int) partitioned by (pdate string) ROW FORMATstring,userid int) partitioned by (pdate string) ROW FORMAT
DELIMITED FIELDS TERMINATED BY ' ';DELIMITED FIELDS TERMINATED BY ' ';
• hive> alter table login add partition(ym='201303',d='28') LOCATIONhive> alter table login add partition(ym='201303',d='28') LOCATION
'hdfs://h46:9000/flume/loginlog/201303/28/';'hdfs://h46:9000/flume/loginlog/201303/28/';

17. Hive UDFHive UDF

18. Hive UDFHive UDF
• StreamingStreaming
• UDFUDF
• UDAFUDAF
• UDTFUDTF

19. streamingstreaming
• 分割字符串分割字符串 pythonpython
• def calcwin():def calcwin():
• for line in sys.stdin:for line in sys.stdin:
• (ldate,userid,roundbet,fold,allin,chipwon) =(ldate,userid,roundbet,fold,allin,chipwon) =
line.strip().split()line.strip().split()
• print 't'.join(["%s:%s"%(ldate,userid),print 't'.join(["%s:%s"%(ldate,userid),
win,fold,allin])win,fold,allin])

20. streamingstreaming
• 用法类似用法类似
• hive> from testpoker selecthive> from testpoker select
transform(ldate,ltime,threadid,gameid,userid,pid,routransform(ldate,ltime,threadid,gameid,userid,pid,rou
ndbet,fold,allin,cardtype,cards,chipwon) usingndbet,fold,allin,cardtype,cards,chipwon) using
'calcpoker.py' as'calcpoker.py' as
(ldate,gameid,userid,pid,win,fold,allin,cardtype,card(ldate,gameid,userid,pid,win,fold,allin,cardtype,card
s) ;s) ;

21. UDFUDF
• public class UDFTest extends UDF {public class UDFTest extends UDF {
• public Integer evaluate(String s) {public Integer evaluate(String s) {
• if (s == null) { return null; }if (s == null) { return null; }
• return s.length(); }return s.length(); }
• }}

22. UDFUDF
• add jar /path/testudf.jar;add jar /path/testudf.jar;
• CREATE TEMPORARY FUNCTION testlength ASCREATE TEMPORARY FUNCTION testlength AS
'org.zhouhh.UDFTest';'org.zhouhh.UDFTest';
• SELECT testlength(src.value) FROM src;SELECT testlength(src.value) FROM src;

23. UDAFUDAF
• User-Defined Aggregation FuncationUser-Defined Aggregation Funcation
• public class UDAFCount extends UDAF {public class UDAFCount extends UDAF {
• public static class Evaluator implements UDAFEvaluator {public static class Evaluator implements UDAFEvaluator {
• private int mCount;private int mCount;
• public void init() { mcount = 0; }public void init() { mcount = 0; }
• public boolean iterate(Object o) {public boolean iterate(Object o) {
• if (o!=null) mCount++;if (o!=null) mCount++;
• return true; }return true; }
• public Integer terminatePartial() {return mCount; }public Integer terminatePartial() {return mCount; }
• public boolean merge(Integer o) {public boolean merge(Integer o) {
• mCount += o;mCount += o; return true;return true; }}
• public Integer terminate() {return mCount; } }public Integer terminate() {return mCount; } }

24. UDAFUDAF
• add jar /path/testudaf.jar;add jar /path/testudaf.jar;
• CREATE TEMPORARY FUNCTION testcount ASCREATE TEMPORARY FUNCTION testcount AS
'org.zhouhh.'org.zhouhh.UDAFCountUDAFCount ';';
• SELECT testcount(src.id) FROM src;SELECT testcount(src.id) FROM src;

25. UDTFUDTF
• User-Defined Table-GeneratingUser-Defined Table-Generating
FunctionsFunctions
• 解决 入一行 出多行输 输解决 入一行 出多行输 输 (On-to-many(On-to-many
maping)maping) 的需求的需求

26. UDTFUDTF
• 承继承继
org.apache.hadoop.hive.ql.udf.generic.GenericUDTorg.apache.hadoop.hive.ql.udf.generic.GenericUDT
FF 。。
• 实现实现 initialize, process, closeinitialize, process, close 三个方法三个方法

27. UDTFUDTF
• 使用方法使用方法
• 1.1. 不可添加其他字段不可添加其他字段 ,, 不可不可 group bygroup by ，， sort bysort by 等等
• select explode_map(properties) as (col1,col2) fromselect explode_map(properties) as (col1,col2) from
src;src;
• 2.2. 用用 lateral viewlateral view
• select src.id, mytable.col1, mytable.col2 from srcselect src.id, mytable.col1, mytable.col2 from src
lateral view explode_map(properties) mytable aslateral view explode_map(properties) mytable as
col1, col2;col1, col2;

28. 小技巧小技巧

29. 小技巧小技巧
• structstruct
• 10158262351015826235
[{"product_id":220003038067,"timestamps":"134032[{"product_id":220003038067,"timestamps":"134032
1132000"},1132000"},
{"product_id":300003861266,"timestamps":"134027{"product_id":300003861266,"timestamps":"134027
1857000"}]1857000"}]

30. 小技巧小技巧
• CREATE EXTERNAL TABLE IF NOT EXISTSCREATE EXTERNAL TABLE IF NOT EXISTS
SampleTableSampleTable
• (
• USER_ID BIGINT,
• NEW_ITEM ARRAY<STRUCT<PRODUCT_ID:
BIGINT,TIMESTAMPS:STRING>>)

31. 小技巧小技巧
• SELECTSELECT
• user_id,
• prod_and_ts.product_id as product_id,
• prod_and_ts.timestamps as timestamps
• FROM
• SampleTable
• LATERAL VIEW explode(new_item)
exploded_table as prod_and_ts;

32. 小技巧小技巧
• **USER_ID** | **PRODUCT_ID** | **TIMESTAMPS****USER_ID** | **PRODUCT_ID** | **TIMESTAMPS**
• ------------+------------------+----------------------------+------------------+----------------
• 1015826235 220003038067 13403211320001015826235 220003038067 1340321132000
• 1015826235 300003861266 13402718570001015826235 300003861266 1340271857000
•
•
•

33. 讨论 ...
谢谢 !
http://abloz.com
2013.4.18
@abloz