Downloaded 25 times
Uploaded byJinho Kim
Tajo Seoul Meetup-201501
This document provides an introduction and quick start guide for Apache Tajo. It outlines how to install Tajo in local or distributed mode, configure basic settings, and launch a Tajo cluster on a single machine or across multiple machines. The document also introduces the speaker and provides contact information.
More Related Content
Tajo Seoul Meetup-201501
- 1. Apache Tajo Quick Start Jinho Kim (jinossy@gmail.com)
- 2. Agenda • Introduc6on to Tajo • Tajo Quick Start • Introduc6on to Text files
- 3. About me • Jinho Kim – Senior Research Engineer, Gruter Corp (2011. 5 ~) – Full-‐Hme contributor to Apache Tajo (2013.6 ~ ) – Apache Tajo PMC member and commiOer (2013.3 ~ ) • Contacts – Email: jhkim AT apache.org – Linkedin: hOp://linkedin.com/in/jinossy/ – TwiOer: @jinossy
- 4.
- 5. Apache Tajo • Open-‐source big data warehouse (also called SQL-‐on-‐hadoop) system • Apache Top-‐level project since March 2014 • Supports SQL standards • Low latency, and long running batch queries • 0.9.0 released in Oct 2014.
- 6. Hadoop eco-‐system Integra6on • De-‐factor standard file format support – Parquet, RCFile, SequenceFile, and Text files • Hcatalog support – Enable Tajo to access exisHng tables used in Hive and others • Yarn support – Tajo can be run on Yarn cluster by using Apache Slider.
- 7. Overall Architecture Master Server (HA) Client JDBC TSql Web UI CatalogStore DBMS HCatalog Submit a Query Manage metadata Allocate a query Send task & monitor Send task & monitor Slave Server TajoWorker QueryMaster Local FileSyst em HDFS Local Query Engine StorageManager Slave Server TajoWorker QueryMaster Local FileSyst em HDFS Local Query Engine StorageManager Slave Server TajoWorker QueryMaster Local FileSyst em HDFS Local Query Engine StorageManager 타조마스터 TajoMaster
- 8.
- 9. 본 문서는개발 브랜치 기준으로 작성 되었으며 문서 작성일 기준으로 공식 릴 리즈 되지 않았습니다. 아래 링크는 미리 준비한 Tarball 입니다. hOp://people.apache.org/~jhkim/tajo-‐0.10.0-‐ SNAPSHOT.tar.gz
- 10. Installing Tajo • Requirement – Linux – JDK 1.6 or 1.7 – Hadoop 2.3.0 or higher • $ wget hOp://archive.apache.org/dist/hadoop/common/ hadoop-‐2.6.0/hadoop-‐2.6.0.tar.gz • $ tar xvzf hadoop-‐2.6.0.tar.gz • From a Release Tarball – $ wget hOp://archive.apache.org/dist/tajo/tajo-‐X.X.X/tajo-‐ X.X.X.tar.gz – $ tar xvzf tajo-‐X.X.X-‐SNAPSHOT.tar.gz
- 11. Installing Tajo • Requirement – Maven 3 – Protocol buffer 2.5.0 • Building from source code – $ git clone hOps://github.com/apache/tajo.git tajo – $ cd tajo – $ mvn clean install -‐Pdist -‐DskipTests –Dtar – $ cp tajo-‐dist/target/tajo-‐X.X.X-‐SNAPSHOT.tar.gz {TAJO_HOME}
- 12. Tajo Cluster Mode • Local mode – A local mode Tajo instance can start up with very simple configuraHons. • Fully distributed mode – A fully distributed mode enables a Tajo instance to run on (HDFS). In this mode, a number of Tajo workers run across a number of the physical nodes where HDFS data nodes run.
- 13. Seng up a Local mode • Local mode Hadoop Cluster 없이
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21. 추천 • SSH • conf/tajo-‐env.sh • Launch a Tajo cluster export HADOOP_HOME={HADOOP_HOME} export JAVA_HOME={JAVA_HOME} export TAJO_WORKER_HEAPSIZE=1000 # export TAJO_LOG_DIR=${TAJO_HOME}/logs $ ssh-‐keygen -‐t rsa $ ssh-‐copy-‐id ~/.ssh/id_rsa.pub {hostname} $ $TAJO_HOME/bin/start-‐tajo.sh
- 22. Seng up a Local mode -‐ Op6onal • tajo.rootdir (tajo-‐site.xml) – warehouse, system 등 데이터
- 23.
- 24. 디렉토리 • tajo.worker.tmpdir.loca6ons (tajo-‐site.xml) – Query 실행에
- 25.
- 26.
- 27.
- 28.
- 29. 디렉토리 property nametajo.rootdir/name valuefile:///tajo/meetup/warehouse/value descripHonBase directory including system directories./descripHon /property property nametajo.worker.tmpdir.locaHons/name value/tmp/tajo-‐${user.name}/tmpdir/value descripHonA base for other temporary directories./descripHon /property
- 30. Seng up a Fully distributed mode • conf/tajo-‐site.xml – Master 와
- 31.
- 32.
- 33.
- 34. RPC 설정 property nametajo.rootdir/name valuehdfs://hostname:port/tajo/value /property property nametajo.master.umbilical-‐rpc.address/name valuehostname:26001/value /property property nametajo.master.client-‐rpc.address/name valuehostname:26002/value /property property nametajo.resource-‐tracker.rpc.address/name valuehostname:26003/value /property property nametajo.catalog.client-‐rpc.address/name valuehostname:26005/value /property
- 35. Seng up a Fully distributed mode • SSH – 모든
- 36.
- 37.
- 38.
- 39. 등록 • Hadoop Home • conf/workers – Worker 로
- 40.
- 41.
- 42.
- 43. 등록 Hostname1 Hostname2 Hostname3 … export HADOOP_HOME={HADOOP_HOME}
- 44. Seng up a Fully distributed mode • Make base directories and set permissions • Distribute a Tajo home to workers • Launch a Tajo cluster $ $TAJO_HOME/bin/start-‐tajo.sh $ $HADOOP_HOME/bin/hadoop fs -‐mkdir /tajo $ $HADOOP_HOME/bin/hadoop fs -‐chmod g+w /tajo
- 45. First query execu6on • Sample data • HDFS (Op6onal) mkdir table1; cat table1/data.csv EOF 1|abc|1.1|a 2|def|2.3|b 3|ghi|3.4|c 4|jkl|4.5|d 5|mno|5.6|e EOF $ $HADOOP_HOME/bin/hadoop fs -‐mkdir /tajo/warehouse/table1 $ $HADOOP_HOME/bin/hadoop fs -‐put data.csv /tajo/warehouse/table1
- 46. First query execu6on • Star6ng the Tajo Shell (tsql) • Tsql usage $ ${TAJO_HOME}/bin/tsql usage: tsql [opHons] [database] -‐B,-‐-‐background execute as background process -‐c,-‐-‐command arg execute only single command, then exit -‐conf,-‐-‐conf arg configuraHon value -‐f,-‐-‐file arg execute commands from file, then exit -‐h,-‐-‐host arg Tajo server host -‐help,-‐-‐help help -‐p,-‐-‐port arg Tajo server port -‐param,-‐-‐param arg parameter value in SQL file
- 47. First query execu6on • Create tables in Tajo – Managed Table – External Table • HDFS 일경우
- 48.
- 49. 변경한다. $ default create table table1 ( id int, name text, score float, type text) using text; $ default create external table table1 ( id int, name text, score float, type text) using text with ('text.delimiter'='|') locaHon 'file:/tajo/meetup/table1';
- 50. First query execu6on • Selec6ng data default select * from table1 where id 2; Progress: 100%, response Hme: 0.492 sec id, name, score, type -‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐ 3, ghi, 3.4, c 4, jkl, 4.5, d 5, mno, 5.6, e (3 rows, 0.492 sec, 36 B selected) default q
- 51. Maximum number of parallel running tasks • Worker Heap Memory Size – Tajo Worker 는
- 52.
- 53.
- 54.
- 55.
- 56.
- 57.
- 58.
- 59. 필요함 • TAJO_WORKER_HEAPSIZE=8000 (8GB) • Worker Resources – CPU Core, Memory Size, Disk 수를
- 60.
- 61.
- 62.
- 63.
- 64.
- 65.
- 66.
- 67. Maximum number of parallel running tasks • Worker Memory Resource – tajo.worker.resource.memory-‐mb • 1개의
- 68.
- 69.
- 70.
- 71.
- 72.
- 73.
- 74.
- 75.
- 76.
- 77.
- 78.
- 79.
- 80.
- 81.
- 82.
- 83.
- 84.
- 85. Maximum number of parallel running tasks • Worker Disk Resource – Disk 설정은
- 86.
- 87.
- 88.
- 89.
- 90.
- 91.
- 92.
- 93.
- 94.
- 95.
- 96.
- 97.
- 98.
- 99.
- 100.
- 101.
- 102.
- 103.
- 104.
- 105.
- 106.
- 107. 정의한다 – tajo.worker.resource.dfs-‐dir-‐aware • Hadoop DataNode 에
- 108.
- 109.
- 110.
- 111.
- 112.
- 113.
- 114.
- 115.
- 116.
- 117. Maximum number of parallel running tasks • 아래의
- 118.
- 119.
- 120.
- 121.
- 122.
- 123.
- 124.
- 125.
- 126. 있다 TAJO_WORKER_HEAPSIZE=8512 property nametajo.worker.resource.memory-‐mb/name value8512/value /property property nametajo.task.memory-‐slot-‐mb.default/name value2000/value /property property nametajo.worker.resource.disks/name value3.0/value /property property nametajo.task.disk-‐slot.default/name value1.0/value /property
- 127. INTRODUCTION TO TEXT FILES
- 128. Text Files • TEXT File Format – 열은
- 129.
- 130.
- 131.
- 132.
- 133.
- 134.
- 135.
- 136.
- 137. 파일 • JSON File Format – (JavaScript Object NotaHon) 열은
- 138.
- 139.
- 140.
- 141.
- 142.
- 143. Text Files • Performance Improvement – Byte Array 대신
- 144.
- 145.
- 146.
- 147.
- 148.
- 149.
- 150. 관리 – Low-‐Level Access (sun.misc.Unsafe 활용) • Off-‐heap Memory 에
- 151.
- 152.
- 153.
- 154.
- 155.
- 156. Text Files • Table File format – TEXT 또는 JSON
- 157.
- 158.
- 159.
- 160.
- 161.
- 162. 사용한다. • create table text_table (id int, name text) using text • create table json_table (id int, name text) using json • Physical Proper6es – File format 별로
- 163.
- 164.
- 165.
- 166.
- 167.
- 168. 있다. • create table text_table (id int, name text) using text with (’text.delimiter'=’|’)
- 169. Text Files • Delimiter – 1개의
- 170.
- 171.
- 172. 인식된다 • Line delimiter: CR, LF, CR+LF – Field 는
- 173. 1byte character 로 구분된다. Default: ‘|’ • CSV: ‘text.delimiter’=‘,’ • TSV: ‘text.delimiter’=‘t’ • Hive default : ’text.delimiter'='u0001’ default create external table table1 ( id int, name text, score float, type text) using text with ('text.delimiter'='|') locaHon '/tajo/meetup/table1';
- 174. Text Files • NULL Value Handling – Text file 의
- 175.
- 176.
- 177.
- 178. 옵션 • Default value 는
- 179. empty string 을 사용 • Hive default: 'text.null'='N’ default create external table table1 ( id int, name text, score float, type text) using text with ('text.delimiter'='|', 'text.null'='N') locaHon 'file:/tajo/meetup/table1';
- 180. Text Files • Supported Codecs • Hadoop 이
- 181.
- 182.
- 183. 사용 (Bzip2 미지원) • 일부
- 184.
- 185.
- 186.
- 187. (snappy) • Table Decompression – 파일의
- 188.
- 189.
- 190.
- 191. 사용된다. • Table Compression – Table data 생성과
- 192.
- 193. 압축한다. • Create table as, insert into org.apache.hadoop.io.compress.DeflateCodec org.apache.hadoop.io.compress.GzipCodec org.apache.hadoop.io.compress.SnappyCodec ..
- 194. Text Files • Table Compression – Managed Table – External Table default create table table2 (id int, name text, score float, type text) using text with ('text.delimiter'='|', 'compression.codec'='org.apache.hadoop.io.compress.DeflateCodec') as select * from table1; $ gzip /tajo/meetup/table3/data.csv $bin/tsql default create external table table3 (id int, name text, score float, type text) using text with ('text.delimiter'='|', 'compression.codec'='org.apache.hadoop.io.compress.GzipCodec') locaHon 'file:/tajo/meetup/table3';
- 195. Text Files • JSON – 1개의
- 196.
- 197.
- 198.
- 199. 인식된다 • Line delimiter: CR, LF, CR+LF – Field 는
- 200.
- 201.
- 202. 구분된다 • Nested Data Structures (지원
- 203. 예정) – Parsing error tolerance • 문법
- 204.
- 205.
- 206.
- 207.
- 208.
- 209.
- 210.
- 211.
- 212.
- 213.
- 214. Text Files • JSON /tajo/meetup/json_table1 default create external table json_table1 ( id int, name text, score float, type text) using json with ('text.error-‐tolerance.max-‐num'='5') locaHon '/tajo/meetup/json_table1'; $ mkdir /tajo/meetup/json_table1 $ cat /tajo/meetup/json_table1/data.json EOF {id:1,name:abc, score:1.1, type:a} {id:2,name:def, score:2.3, type:b} {id:3,name:ghi, score:3.4, type:c} {id:4,name:jkl, score:4.5, type:d} {id:5,name:emiya muljomdao,score:5.6, type:e} EOF
- 215. Get Involved! • We are recruiHng contributors! • General – hOp://tajo.apache.org • Ge„ng Started – hOp://tajo.apache.org/docs/0.9.0/ge„ng_started.html • Downloads – hOp://tajo.apache.org/docs/0.9.0/ge„ng_started/downloading_source.html • Jira – Issue Tracker – hOps://issues.apache.org/jira/browse/TAJO • Join the mailing list – dev-‐subscribe@tajo.apache.org – issues-‐subscribe@tajo.apache.org