This document discusses setting up and using Tajo, an Apache Hadoop-based data warehousing system, on AWS. It provides instructions on using Tajo Cloud to easily configure a Tajo cluster on AWS. Examples show how to connect external data from S3, perform queries, and analyze customer cohort data to understand purchase patterns over time. Tajo allows direct access to data in S3 and dynamic scaling of worker nodes, and its connector enables remote querying from SQL clients, Excel, and R.

1. AWS에서Tajo를이용한빅데이터분석실습
고영경
ykko@gruter.com

2. ©2014 Gruter. All rights reserved.
실습안내
AWS Cloud 환경에서Tajo 클러스터를구성하고
Tajo 쿼리를이용하여데이터분석을직접수행해봅니다.
1.Tajo Cloud를이용하여Tajo 클러스터구성하기
2.S3 의데이터를External Table 로연결하기
3.Tajo Connector를이용하여원격질의수행하기
4.갂단한Cohort 분석예제
•실습안내페이지: http://techday.gruter.com:2014/

3. ©2014 Gruter. All rights reserved.
Tajo란?
•Tajo
–하둡기반의대용량데이터웨어하우스시스템
–2010년부터리서치프로토타입으로개발시작
–아파치탑레벨프로젝트
•Features
–SQL 표준호홖
–질의젂체를분산처리
–HDFS가기본스토리지
–관계형모델(Nested model로확장논의중)
–긴시갂이요하는ETL뿐만아니라low-latency 질의를함께지원
•Tajo on AWS
–Hadoop없이동작가능. (EMR에서실행도가능)
–S3 에저장된데이터를바로액세스(로컬HDFS로복사할필요없음)

4. ©2014 Gruter. All rights reserved.
AWS에서Tajo 클러스터셋업하기
•Tajo를다운받아직접설치하셔도됩니다
•그루터의Tajo Cloud AMI 를이용하면더쉽습니다
–최신버젂의Tajo 탑재
–복잡한설정필요없이클릭몇번으로셋업완료
–AWS 홖경에최적화
–그루터개발추가기능(Tajo Proxy, Tajo Connecter, SQL Workbench)
–그루터의기술지원
–AWS 마켓플레이스에롞치준비중
•Tajo Cloud AMI를이용한클러스터셋업
–방법1. AWS Web Console 이용
–방법2. AWS에서CLI (커맨드라인) API이용
–방법3. 그루터의Tajo Cloud 서비스이용
4

5. ©2014 Gruter. All rights reserved.
Tajo Cloud on AWS

6. ©2014 Gruter. All rights reserved.
Tajo Cloud를이용한클러스터셋업(실습)
6http://taas.gruter.com/접속
AWS Account 입력
클러스터설정
클러스터구동완료
1
2
3
4
Region: US East (Virginia)
인스턴스: c3.xlarge, 2 worker 노드
Keypair: demo2014
실습용개인계정입력

7. ©2014 Gruter. All rights reserved.
Tajo 클러스터셋업–Security Group 설정
•AWS Console > EC2 > Security Group >
•“taas” security group > Inbound
7
Protocol
Port
Source
tcp
22
SSH접속을허용할IP
tcp
80
WebWorkbench 접속을허용할IP
tcp
26080
Tajo 관리UI에접속할IP
tcp
26992
Tajo Connector에접속할IP

8. ©2014 Gruter. All rights reserved.
Tajo 사용실습-1
1.Tajo master 노드SSH 접속
2.TSQL (Tajo interactive shell) 실행
3.자신의database 생성
8
$ ssh -i demo2014.pemec2-user@your-master-node-ip
$ sudo su -tajo
$ /home/tajo/tajo/bin/tsql
Try ? for help.
default>
default> CREATE DATABASE db_name; --use your ID as DB name
default> c db_name;
You are now connected to database “db_name" as user "tajo".

9. ©2014 Gruter. All rights reserved.
Tajo 사용실습-2
3. S3 데이터연결
9
CREATE EXTERNAL TABLE orders (
O_ORDERKEY bigint, O_CUSTKEY bigint, O_ORDERSTATUS text,
O_TOTALPRICE double, O_ORDERDATE text, O_ORDERPRIORITY text,
O_CLERK text, O_SHIPPRIORITY int, O_COMMENT text)
USING csvwith ('csvfile.delimiter'='|')
LOCATION 's3n://taas-bucket-us-east-1- 1594485745/tajo/sampleData/tpch-1g/orders';
SELECT * FROM orders LIMIT 10;
s3n://taas-bucket-us-east-1-1594485745/tajo/sampleData/tpch-1g/orders/orders.tbl

10. ©2014 Gruter. All rights reserved.
분석예제–Cohort Analysis (1)
•Cohort 분석
–동일한특성을가짂고객들을그룹(Cohort)로묶어
–시갂의흐름에따라각그룹의성과(유지율, 사용량, 고객가치등)를측정하고비교하는분석방법
10
* 출처: 하용호“스타트업은데이터를어떻게바라봐야할까?” http://www.slideshare.net/yongho/ss-32267675

11. ©2014 Gruter. All rights reserved.
분석예제–Cohort Analysis (2)
•이예제에서는
–TPC-H 샘플데이터의orders 테이블에서
–특정월에첫구매한사용자들을Cohort 로묶고
–각그룹의이후월단위재구매패턴을비교
11
Cohort
첫구매월
1달후
2달후
3달후
4달후
5달후
총합계
1월첫구매그룹
151,292
151,330
150,063
149,407
149,510
152,193
903,795
2월첫구매그룹
150,624
153,407
151,847
148,187
149,797
753,862
3월첫구매그룹
150,328
152,783
149,548
154,045
606,704
4월첫구매그룹
151,178
149,859
148,542
449,579
5월첫구매그룹
152,174
150,412
302,586
6월첫구매그룹
151,265
151,265
총합계
151,292
301,954
453,798
605,215
749,278
906,254
3,167,791
Column
설명
o_orderkey
주문번호
o_custkey
고객번호
o_totalprice
주문금액
o_orderdate
주문일자
…
Table: orders

12. ©2014 Gruter. All rights reserved.
분석예제–Cohort Analysis (3)
•Cohort 구하기
•Cohort 정의: 특정월에첫구매한사용자그룹
12
CREATE TABLE cohort AS
SELECT o_custkey, --고객번호
min(o_orderdate) as cohort_date, --최초주문일
min(substr(o_orderdate,0,8)) as cohort --cohort 그룹
FROM orders
WHERE o_orderdatebetween '1992-01-01' and '1992-06-30'
GROUP BY o_custkey
ORDER BY o_custkey;
•Tajo Cloud 의SQL Workbench 에서실행해보세요

13. ©2014 Gruter. All rights reserved.
SQL Workbench
•http://tajo-master-ip/
•Tajo Cloud 클러스터목록> ACTION > SQL Workbench
•Tip. 설정메뉴에Sample Data (TPC-H 1G) 로드기능이포함되어있음
13

14. ©2014 Gruter. All rights reserved.
분석예제–Cohort Analysis (4)
•각Cohort 의월별재구매계산
14--cohort, 주문월, 주문자수, 주문건수, 주문총액, 평균주문액CREATE TABLE cohort_analysisAS SELECT c.cohort, substr(o_orderdate,0,8) as order_month, count(distinct(o.o_custkey)) as buyer_cnt, count(o.o_orderkey) as order_cnt, round(sum(o.o_totalprice)) as amount, round(avg(o.o_totalprice)) as avg_amountFROM orders o JOIN cohort c ON o.o_custkey= c.o_custkeyWHERE o.o_orderdatebetween '1992-01-01' and '1992-06-30' GROUP BY c.cohort, substr(o_orderdate,0,8) ORDER BY c.cohort, substr(o_orderdate,0,8) ASC
•쿼리실행상황을Tajo 관리UI에서확인해보세요

15. ©2014 Gruter. All rights reserved.
Tajo 관리UI
•http://tajo-master-ip:26080/
•Tajo Cloud 클러스터목록> ACTION > Tajo Master
•Tip. Security Group 설정에서26080 포트오픈필요
15

16. ©2014 Gruter. All rights reserved.
분석예제–Cohort Analysis (5)
16--cohort, 주문월, 평균주문액CREATE TABLE cohort_analysisAS SELECT c.cohort, substr(o_orderdate,0,8) as order_month, round(avg(o.o_totalprice)) as avg_amountFROM orders o JOIN ( SELECT o_custkey, min(o_orderdate) as cohort_date, min(substr(o_orderdate,0,8)) as cohortFROM orders WHERE o_orderdatebetween '1992-01-01' and '1992-06-30' GROUP BY o_custkey) c ON o.o_custkey= c.o_custkeyWHERE o.o_orderdatebetween '1992-01-01' and '1992-06-30' GROUP BY c.cohort, substr(o_orderdate,0,8) ORDER BY c.cohort, substr(o_orderdate,0,8) ASC
•Sub Query 로합쳐보면
•외부SQL툴에서Tajo connector를이용해원격실행해보세요

17. ©2014 Gruter. All rights reserved.
Tajo Connector를이용한원격연결(시연)
•Custom JDBC driver를지원하는툴(SQuirrelSQL, DB Visualizer등)
•TajoCloud 에포함된Proxy 서버를통해연결(26992 포트오픈필요)
•jdbc:taas-tajo://tajo_master_node_ip:26992/db_name
17

18. ©2014 Gruter. All rights reserved.
18

19. ©2014 Gruter. All rights reserved.
분석예제–Cohort Analysis (5)
19
Cohort
1992-01
1992-02
1992-03
1992-04
1992-05
1992-06
총합계
1992-01
151,292
151,330
150,063
149,407
149,510
152,193
903,795
1992-02
150,624
153,407
151,847
148,187
149,797
753,862
1992-03
150,328
152,783
149,548
154,045
606,704
1992-04
151,178
149,859
148,542
449,579
1992-05
152,174
150,412
302,586
1992-06
151,265
151,265
총합계
151,292
301,954
453,798
605,215
749,278
906,254
3,167,791
Cohort
첫구매월
1달후
2달후
3달후
4달후
5달후
총합계
1월첫구매그룹
151,292
151,330
150,063
149,407
149,510
152,193
903,795
2월첫구매그룹
150,624
153,407
151,847
148,187
149,797
753,862
3월첫구매그룹
150,328
152,783
149,548
154,045
606,704
4월첫구매그룹
151,178
149,859
148,542
449,579
5월첫구매그룹
152,174
150,412
302,586
6월첫구매그룹
151,265
151,265 첫구매
재구매

20. ©2014 Gruter. All rights reserved.
분석예제–Cohort Analysis (6)
20144,000
146,000
148,000 150,000 152,000 154,000
156,000
첫구매월1달후2달후
3달후
4달후
5달후
1월첫구매그룹2월첫구매그룹3월첫구매그룹
Cohort
첫구매월
1달후
2달후
3달후
4달후
5달후
총합계
1월첫구매그룹
151,292
151,330
150,063
149,407
149,510
152,193
903,795
2월첫구매그룹
150,624
153,407
151,847
148,187
149,797
753,862
3월첫구매그룹
150,328
152,783
149,548
154,045
606,704
4월첫구매그룹
151,178
149,859
148,542
449,579
5월첫구매그룹
152,174
150,412
302,586
6월첫구매그룹
151,265
151,265

21. ©2014 Gruter. All rights reserved.
Wrap-up
•Tajo Cloud 서비스를이용해서자신만의Tajo cluster 를AWS에쉽게만들수있다.
•Tajo 는S3에저장된데이터를직접액세스할수있다.
•클러스터의워커노드수를동적으로조정할수있다.
•Tajo Connector를이용하여, SQL client, Excel, R, BI 등외부툴과연동할수있다.
•보다자세한내용은taas.gruter.com 의가이드문서를참고하세요
•가장중요한일! 작업이끝나면클러스터를꼭종료하세요.
21

22. ©2014 Gruter. All rights reserved.
Q&A
22

23. ©2014 Gruter. All rights reserved.
Reference
•Apache Tajo Project Home: http://tajo.apache.org
•Tajo Cloud Site : http://taas.gruter.com
•Tajo Cloud User Guide: https://s3-us-west- 2.amazonaws.com/tajo/taas/documents/TaaS_UserGuide.pdf
•Tajo SQL Language Reference: http://tajo.apache.org/docs/current/index.html
•AWS Getting Started: http://aws.amazon.com/ko/documentation/gettingstarted/
23

24. ©2014 Gruter. All rights reserved.
GRUTER: YOUR PARTNER
IN THE BIG DATA REVOLUTION
Phone +82-70-8129-2950
Fax+82-70-8129-2952
E-mail contact@gruter.com
Webwww.gruter.com
Phone +1-415-841-3345