Phd tutorial hawq_v0.1

1. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103

3. Pivotal HAWQ ON-HAND

4. TUTORIAL(1)

21. Prepared By Pivotal Korea

24. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 DOCUMENT CONTROL For any questions regarding this document contact: Name: Seungdon Choi E-mail: schoi@pivotal.io Document Revision History Date Version Description Author Reviewer 01/02/2015 0.1 Draft for internal review 03/02/2015 0.9 For Distribution

25. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 Table of Contents DOCUMENT

26. OVERVIEW ...................................................................................... 3 PREREQUISITE ...................................................................................................... 4 [예제]HAWQ

27. Internal

28. Table.................................................................................. 5 [예제]pxf

29. Table

31. HDFS.......................................................................................... 8 [예제]pxf

32. Table

34. HIVE......................................................................................... 12 DOCUMENT

35. OVERVIEW

37. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 Big Data 와 Hadoop 이 계속 시장에서 화두가 되고 있으며, 인터넷 기업뿐만 아니라 기업환경에서도 Hadoop 이 새로운 데이터 소스와 인프라로서 자리잡아 가고 있다. 하지만 운영과 분석을 위해서는 Map Reduce 등 새로운 기술을 배워야 하는 learning curve 들로 인해서 많은 기업들이 도입을 두려워 하고 있는 것도 현실이다. Pivotal 의 HAWQ 제품은 SQL-On-Hadoop 제품으로 기존에 사용자와 개발자가 익숙한 SQL 인터페이스를 제공하여 Hadoop 의 workload 를 기존의 Data Warehouse 처럼 쉽고 빠르게 빅데이터 프로젝트를 수행할 수 있게 한다. 본 문서는 Pivotal HD Single Node VM 을 사용하여 HAWQ 의 기본 사용법에 대한 Hands On Practice 를 다룬다.

38. 참고

39. 문서

42. Pivotal HAWQ 개발가이드 Pivotal HD 2.1.0 Documentation: http://pivotalhd.docs.pivotal.io/doc/2100/index.html HAWQ Administration Guide: http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/index.html#topics/HAWQAdminist ration.html PXF Guide: http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/index.html#topics/PivotalExtensio nFrameworkPXF.html Getting Started Data Tutorial: http://pivotalhd.docs.pivotal.io/tutorial/getting-started/overview.html

46. PREREQUISITE

48. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 실습에서 사용될 Data Set 과 샘플 스크립트를 준비한다. (1)테스트를 위한 Single node VM 을 다운받아 수행시킨다. 자세한 내용은 https://pivotalkr.wordpress.com/2015/01/02/phd-single-node-vm 활용기 1-vm- 이미지-다운로드-sample-code/ 을 참고한다. (2)샘플 데이터를 하기의 URL 에서 다운받아 VM 에 업로드한다. https://github.com/gopivotal/pivotal-samples/tree/master/sample-data [pivhdsne:~]$ ls retail_demo categories_dim.tsv.gz date_dim.tsv.gz orders.tsv.gz customer_addresses_dim.tsv.gz email_addresses_dim.tsv.gz payment_methods.tsv.gz customers_dim.tsv.gz order_lineitems.tsv.gz products_dim.tsv.gz HDFS 에 상기 file 들을 load 한다. #hadoop fs –put retail_demo/ / 혹은 sh ./load_data_to_HDFS.sh 정상적으로 file 들이 load 됨을 확인한다. Hadoop fsck /retail_demo [pivhdsne:~]$ hadoop fs -ls /retail_demo Found 9 items -rw-r--r-- 3 gpadmin hadoop 590 2015-01-30 14:50 /retail_demo/categories_dim.tsv.gz -rw-r--r-- 3 gpadmin hadoop 53995977 2015-01-30 14:50 /retail_demo/customer_addresses_dim.tsv.gz -rw-r--r-- 3 gpadmin hadoop 4646775 2015-01-30 14:50 /retail_demo/customers_dim.tsv.gz -rw-r--r-- 3 gpadmin hadoop 17772 2015-01-30 14:50 /retail_demo/date_dim.tsv.gz -rw-r--r-- 3 gpadmin hadoop 7760971 2015-01-30 14:50 /retail_demo/email_addresses_dim.tsv.gz -rw-r--r-- 3 gpadmin hadoop 137780165 2015-01-30 14:50 /retail_demo/order_lineitems.tsv.gz -rw-r--r-- 3 gpadmin hadoop 72797064 2015-01-30 14:50 /retail_demo/orders.tsv.gz -rw-r--r-- 3 gpadmin hadoop 99 2015-01-30 14:50 /retail_demo/payment_methods.tsv.gz -rw-r--r-- 3 gpadmin hadoop 23333203 2015-01-30 14:50 /retail_demo/products_dim.tsv.gz [예제]HAWQ

49. INTERNAL

50. TABLE

53. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 HAWQ 의 internal table 을 생성하고, Sample 데이터를 로드하도록 하자. 우선 기존에 있는 retail_demo schema 를 삭제 후 재생성한다. [pivhdsne:hawq_tables]$ psql psql (8.2.15) Type help for help. gpadmin=# drop schema retail_demo; [pivhdsne:hawq_tables]$ psql psql (8.2.15) Type help for help. gpadmin=# drop schema retail_demo; gpadmin=# i /pivotal-samples/hawq/hawq_tables/create_hawq_tables.sql DROP TABLE CREATE TABLE DROP TABLE CREATE TABLE DROP TABLE CREATE TABLE DROP TABLE CREATE TABLE DROP TABLE CREATE TABLE DROP TABLE CREATE TABLE DROP TABLE CREATE TABLE DROP TABLE CREATE TABLE DROP TABLE CREATE TABLE Sample Data 를 copy 명령어를 사용하여 각 HAWQ table 에 load 한다. [pivhdsne:hawq_tables]$ cd /home/gpadmin/retail_demo/ [pivhdsne:retail_demo]$ ls -lrt total 293632

54. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 -rw-r--r-- 1 gpadmin gpadmin 590 Jan 30 14:42 categories_dim.tsv.gz -rw-r--r-- 1 gpadmin gpadmin 7760971 Jan 30 14:42 email_addresses_dim.tsv.gz -rw-r--r-- 1 gpadmin gpadmin 17772 Jan 30 14:42 date_dim.tsv.gz -rw-r--r-- 1 gpadmin gpadmin 4646775 Jan 30 14:42 customers_dim.tsv.gz -rw-r--r-- 1 gpadmin gpadmin 53995977 Jan 30 14:42 customer_addresses_dim.tsv.gz -rw-r--r-- 1 gpadmin gpadmin 137780165 Jan 30 14:42 order_lineitems.tsv.gz -rw-r--r-- 1 gpadmin gpadmin 23333203 Jan 30 14:42 products_dim.tsv.gz -rw-r--r-- 1 gpadmin gpadmin 99 Jan 30 14:42 payment_methods.tsv.gz -rw-r--r-- 1 gpadmin gpadmin 72797064 Jan 30 14:42 orders.tsv.gz zcat customers_dim.tsv.gz | psql -c COPY retail_demo.customers_dim_hawq FROM STDIN DELIMITER E't' NULL E''; zcat categories_dim.tsv.gz | psql -c COPY retail_demo.categories_dim_hawq FROM STDIN DELIMITER E't' NULL E''; zcat order_lineitems.tsv.gz | psql -c COPY retail_demo.order_lineitems_hawq FROM STDIN DELIMITER E't' NULL E''; zcat orders.tsv.gz | psql -c COPY retail_demo.orders_hawq FROM STDIN DELIMITER E't' NULL E''; zcat customer_addresses_dim.tsv.gz | psql -c COPY retail_demo.customer_addresses_dim_hawq FROM STDIN DELIMITER E't' NULL E''; zcat email_addresses_dim.tsv.gz | psql -c COPY retail_demo.email_addresses_dim_hawq FROM STDIN DELIMITER E't' NULL E''; zcat products_dim.tsv.gz | psql -c COPY retail_demo.products_dim_hawq FROM STDIN DELIMITER E't' NULL E''; zcat payment_methods.tsv.gz | psql -c COPY retail_demo.payment_methods_hawq FROM STDIN DELIMITER E't' NULL E''; zcat date_dim.tsv.gz | psql -c COPY retail_demo.date_dim_hawq FROM STDIN DELIMITER E't' NULL E''; 데이터가 정상적으로 load 되었는지 확인하자. [pivhdsne:hawq_tables]$ pwd /pivotal-samples/hawq/hawq_tables [pivhdsne:hawq_tables]$ sh ./verify_load_hawq_tables.sh Table Name | Count -----------------------------+------------------------ customers_dim_hawq | 401430 categories_dim_hawq | 56 customer_addresses_dim_hawq | 1130639 email_addresses_dim_hawq | 401430 order_lineitems_hawq | 1024158 orders_hawq | 512071 payment_methods_hawq | 5 products_dim_hawq | 698911 -----------------------------+------------------------ HAWQ 는 기본적으로 Greenplum 엔진(postgresql 8.2 를 기반으로 개발된 MPP SQL 엔진) 을 그대로 하둡 HDFS 에서 구현한 SQL on Hadoop 엔진이므로, Greenplum/Postgresql 에서 사용하던 SQL 구문을 그대로 사용할 수 있다.

55. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 HAWQ 에 query 를 날려보도록 하자. Order table 에서 각 우편번호별 총 지급액, 총세액을 구하는 query 이다. [pivhdsne:hawq_tables]$ psql psql (8.2.15) Type help for help. gpadmin=# select billing_address_postal_code, sum(total_paid_amount::float8) as total, sum(total_tax_amount::float8) as tax from retail_demo.orders_hawq group by billing_address_postal_code order by total desc limit 10; billing_address_postal_code | total | tax -----------------------------+-----------+----------- 48001 | 111868.32 | 6712.0992 15329 | 107958.24 | 6477.4944 42714 | 103244.58 | 6194.6748 41030 | 101365.5 | 6081.93 50223 | 100511.64 | 6030.6984 03106 | 83566.41 | 0 57104 | 77383.63 | 3095.3452 23002 | 73673.66 | 3683.683 25703 | 68282.12 | 4096.9272 26178 | 66836.4 | 4010.184 (10 rows) gpadmin=# [예제]PXF

56. TABLE

58. HDFS

59. 여기서는 HAWQ 의 PXF External Table 을 생성하는 예제를 설명한다. HAWQ PXF External Table 을 사용하면 Pivotal HD 상의 다양한 native format(comma separated, tab delimited, plain text file 등) 으로 정의된 dataset 을 읽고 쓸 수 있다. Load 할 data set 을 확인하자. [pivhdsne:retail_demo]$ hadoop fs -ls /retail_demo Found 9 items drwxr-xr-x - gpadmin hadoop 0 2015-02-02 13:45 /retail_demo/categories_dim

60. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 drwxr-xr-x - gpadmin hadoop 0 2015-02-02 13:45 /retail_demo/customer_addresses_dim drwxr-xr-x - gpadmin hadoop 0 2015-02-02 13:45 /retail_demo/customers_dim drwxr-xr-x - gpadmin hadoop 0 2015-02-02 13:45 /retail_demo/date_dim drwxr-xr-x - gpadmin hadoop 0 2015-02-02 13:45 /retail_demo/email_addresses_dim drwxr-xr-x - gpadmin hadoop 0 2015-02-02 13:45 /retail_demo/order_lineitems drwxr-xr-x - gpadmin hadoop 0 2015-02-02 13:45 /retail_demo/orders drwxr-xr-x - gpadmin hadoop 0 2015-02-02 13:45 /retail_demo/payment_methods drwxr-xr-x - gpadmin hadoop 0 2015-02-02 13:45 /retail_demo/products_dim [pivhdsne:retail_demo]$ hadoop fs -ls /retail_demo/categories_dim Found 1 items -rw-r--r-- 3 gpadmin hadoop 590 2015-02-02 13:45 /retail_demo/categories_dim/categories_dim.tsv.gz 데이터 확인 hadoop fs -cat /retail_demo/categories_dim.tsv.gz |zcat External Table 을 생성한다. ! 이 query 로 수행하면 Fragmenter deprecate 가 되어 워닝이 난다. http://pivotalhd.docs.pivotal.io/tutorial/getting-started/hawq/pxf- external-tables.html 에 있는 External Table Creation 명령어를 이용하자. [pivhdsne:pxf_tables]$ pwd /pivotal-samples/hawq/pxf_tables [pivhdsne:pxf_tables]$ psql psql (8.2.15) Type help for help. gpadmin=# i create_pxf_tables.sql External Table 의 문법을 확인해보자. 미리 정의되어 있는 Profile – HdfsTextSimple 을 사용하여 정의한다. 기정의되어 있는 Profile List 는 http://pivotalhd.docs.pivotal.io/doc/2100/webhelp/index.html#topics/PXFInsta llationandAdministration.html 에서 확인할 수 있다. CREATE EXTERNAL TABLE retail_demo.payment_methods_pxf ( payment_method_id smallint, payment_method_code character varying(20) ) LOCATION ('pxf://pivhdsne:50070/retail_demo/payment_methods/payment_methods.tsv.gz?profile=HdfsTextSimple') FORMAT 'TEXT' (DELIMITER = E't');

62. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 -----------------------------+-----------+----------- 48001 | 111868.32 | 6712.0992 15329 | 107958.24 | 6477.4944 42714 | 103244.58 | 6194.6748 41030 | 101365.5 | 6081.93 50223 | 100511.64 | 6030.6984 03106 | 83566.41 | 0 57104 | 77383.63 | 3095.3452 23002 | 73673.66 | 3683.683 25703 | 68282.12 | 4096.9272 26178 | 66836.4 | 4010.184 (10 rows) 통계정보의 생성 PXF External Table 의 경우도 하기의 예제와 같이 통계정보를 수집하여 SQL 수행시 최적의 query plan 을 작성하는데 도움을 줄 수 있다. [pivhdsne:~]$ seq 1 10000000 /tmp/demo.txt 셈플 데이터 생성 [pivhdsne:~]$ hadoop fs -put /tmp/demo.txt / Hadoop 에 load [pivhdsne:~]$ psql psql (8.2.15) Type help for help. gpadmin=# timing Timing is on. gpadmin=# CREATE EXTERNAL TABLE demo (val INT) External Table 생성 gpadmin-# LOCATION ('pxf://pivhdsne:50070/demo.txt?Fragmenter=com.pivotal.pxf.plugins.hdfs.HdfsDataFragmenterAnal yzer=com.pivotal.pxf.plugins.hdfs.HdfsAnalyzerAccessor=com.pivotal.pxf.plugins.hdfs.TextFileAccess orResolver=com.pivotal.pxf.plugins.hdfs.TextResolver') gpadmin-# FORMAT 'TEXT' (DELIMITER = '|'); CREATE EXTERNAL TABLE Time: 52.876 ms gpadmin=# select relpages,reltuples from pg_class where relname='demo'; 통계정보 확인 relpages | reltuples ----------+----------- 1000 | 1e+06 (1 row)

63. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 Time: 137.892 ms gpadmin=# select val from demo where val=59999; query 수행 val ------- 59999 (1 row) Time: 3840.291 ms gpadmin=# analyze demo; 통계정보 수행 ANALYZE Time: 258.789 ms gpadmin=# select relpages,reltuples from pg_class where relname='demo'; relpages | reltuples ----------+----------- 4096 | 161858 (1 row) Time: 101.710 ms gpadmin=# select val from demo where val=59999; val ------- 59999 (1 row) Time: 2734.797 ms gpadmin=# [예제]PXF

64. TABLE

66. HIVE

67. 간단한 HDFS plain text 뿐만 아니라 PXF 를 사용하여 Hive,HBase 같은 소스도 External Table 로 access 가 가능하다. (1) Hive table 생성

68. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 Hive Table 을 생성하고, $ hive hive CREATE DATABASE IF NOT EXISTS retail_demo; USE retail_demo; CREATE TABLE retail_demo.order_lineitems_hive ( Order_ID string , Order_Item_ID bigint , Product_ID int , Product_Name string , Customer_ID int , Store_ID int , Item_Shipment_Status_Code string , Order_Datetime timestamp , Ship_Datetime timestamp , Item_Return_Datetime timestamp , Item_Refund_Datetime timestamp , Product_Category_ID int , Product_Category_Name string , Payment_Method_Code string , Tax_Amount double , Item_Quantity int , Item_Price double , Discount_Amount double , Coupon_Code string , Coupon_Amount double , Ship_Address_Line1 string , Ship_Address_Line2 string , Ship_Address_Line3 string , Ship_Address_City string , Ship_Address_State string , Ship_Address_Postal_Code string , Ship_Address_Country string , Ship_Phone_Number string , Ship_Customer_Name string , Ship_Customer_Email_Address string , Ordering_Session_ID string , Website_URL string

69. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 ) -- PARTITIONED BY (Order_Datetime timestamp) ROW FORMAT DELIMITED FIELDS TERMINATED BY 't' STORED AS TEXTFILE LOCATION '/retail_demo/order_lineitems/'; hive select count(*) from retail_demo.customers_dim_hive; Total MapReduce jobs = 1 .. 2 seconds 940 msec Ended Job = job_1370914856264_0009 MapReduce Jobs Launched: Job 0: Map: 1 Reduce: 1 Cumulative CPU: 2.94 sec HDFS Read: 4646997 HDFS Write: 7 SUCCESS Total MapReduce CPU Time Spent: 2 seconds 940 msec OK 401430 Time taken: 20.03 seconds (2) HAWQ 에서 External Table 을 만들어서 이 Hive Table 을 읽어보자. [pivhdsne:~]$ psql psql (8.2.15) Type help for help CREATE EXTERNAL TABLE retail_demo.order_lineitems_hive ( Order_ID text , Order_Item_ID bigint , Product_ID int , Product_Name text , Customer_ID int , Store_ID int , Item_Shipment_Status_Code text , Order_Datetime timestamp , Ship_Datetime timestamp , Item_Return_Datetime timestamp , Item_Refund_Datetime timestamp , Product_Category_ID int , Product_Category_Name text , Payment_Method_Code text

70. pivotal.io 875 Howard Street, Fifth Floor, San Francisco, CA 94103 , Tax_Amount float8 , Item_Quantity int , Item_Price float8 , Discount_Amount float8 , Coupon_Code text , Coupon_Amount float8 , Ship_Address_Line1 text , Ship_Address_Line2 text , Ship_Address_Line3 text , Ship_Address_City text , Ship_Address_State text , Ship_Address_Postal_Code text , Ship_Address_Country text , Ship_Phone_Number text , Ship_Customer_Name text , Ship_Customer_Email_Address text , Ordering_Session_ID text , Website_URL text ) LOCATION ('pxf://pivhdsne:50070/retail_demo.order_lineitems_hive?PROFILE=hive') FORMAT 'CUSTOM' (formatter='pxfwritable_import'); gpadmin=# select count(*) from retail_demo.order_lineitems_hive; count --------- 1024158 (1 row)

Phd tutorial hawq_v0.1

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (20)

Similar to Phd tutorial hawq_v0.1

Similar to Phd tutorial hawq_v0.1 (20)

Recently uploaded

Recently uploaded (20)

Phd tutorial hawq_v0.1