Replacing Telco DB/DW to Hadoop and Hive

Replacing Telco DB/DW to Hadoop and Hive JunHo Cho Data Analysis Platform TeamFriday, July 1, 2011

• Cloud Computing Platform - Xen • Cloud Storage Platform - hadoop • Massive Email Archiving Solution - hadoop, lucene • HIVE : social network analysis using email • Log Archiving Solution - hadoop • Data Analysis data mining, machine learning, data statistic • Data Platform - hadoop, lucene, hive • Cloud Architecture - KT CloudFriday, July 1, 2011

Telco DataFriday, July 1, 2011

Friday, July 1, 2011

OpenSourceFriday, July 1, 2011

OpenSource Storage & ComputingFriday, July 1, 2011

OpenSource CollectionFriday, July 1, 2011

OpenSource SearchFriday, July 1, 2011

OpenSource AnalysisFriday, July 1, 2011

OpenSource CoordinationFriday, July 1, 2011

Hive InternalFriday, July 1, 2011

Hive Architecture UI Driver DDL HQL Execution Works Engine MetaStore Compiler ORM Hadoop ResultFriday, July 1, 2011

Hive Architecture UI Driver select col1 from tab1 where ... DDL HQL Execution Works Engine MetaStore Compiler ORM Hadoop ResultFriday, July 1, 2011

Hive Architecture UI Driver DDL HQL Execution Works Engine MetaStore Compiler ORM Hadoop ResultFriday, July 1, 2011

Hive Architecture a 123344 b 121211 c 342434 UI Driver DDL HQL Execution Works Engine MetaStore Compiler ORM Hadoop ResultFriday, July 1, 2011

Hive Internal Map Reduce Web UI Hive CLI JDBC TSOperator User Script Browse, Query, DDL UDF/UDAF SELOperator substr sum MetaStore Hive QL FSOperator average Thrift API Parser ExecMapper/ExecReducer Plan SerDe Optimizer Input/OutputFormat Task HDFS StorageHandler RCFile DB ... HBaseFriday, July 1, 2011

Parser Parser Select col1,col2 From tab1 Where col3 > 5 TOK_QUERY TOK_FROM TOK_INSERT TOK_DESTINATION TOK_SELECT TOK_WHERE TOK_TABNAME TOK_SELEXPR TOK_SELEXPR TOK_DIR > TOK_TABLE_OR_COL TOK_TABLE_OR_COL TOK_TMP_FILE TOK_TABLE_OR_COL 5Friday, July 1, 2011

Parser Parser Select col1,col2 From tab1 Where col3 > 5 QB TOK_QUERY TOK_FROM TOK_INSERT TOK_DESTINATION TOK_SELECT TOK_WHERE TOK_TABNAME TOK_SELEXPR TOK_SELEXPR TOK_DIR > TOK_TABLE_OR_COL TOK_TABLE_OR_COL TOK_TMP_FILE TOK_TABLE_OR_COL 5Friday, July 1, 2011

Parser Parser Select col1,col2 From tab1 Where col3 > 5 TOK_QUERY TOK_FROM TOK_INSERT TOK_DESTINATION TOK_SELECT TOK_WHERE TOK_TABNAME TOK_SELEXPR TOK_SELEXPR QB tab1 TOK_DIR > TOK_TABLE_OR_COL TOK_TABLE_OR_COL TOK_TMP_FILE TOK_TABLE_OR_COL 5Friday, July 1, 2011

Parser Parser Select col1,col2 From tab1 Where col3 > 5 TOK_QUERY TOK_FROM TOK_INSERT TOK_DESTINATION TOK_SELECT TOK_WHERE TOK_TABNAME TOK_SELEXPR TOK_SELEXPR tab1 TOK_DIR > TOK_TABLE_OR_COL TOK_TABLE_OR_COL TOK_TMP_FILE TOK_TABLE_OR_COL 5 QB insclause-0Friday, July 1, 2011

Parser Parser Select col1,col2 From tab1 Where col3 > 5 TOK_QUERY TOK_FROM TOK_INSERT TOK_DESTINATION TOK_SELECT TOK_WHERE TOK_TABNAME TOK_SELEXPR TOK_SELEXPR tab1 TOK_DIR > TOK_TABLE_OR_COL TOK_TABLE_OR_COL TOK_TMP_FILE col1 QB TOK_TABLE_OR_COL 5 insclause-0Friday, July 1, 2011

Parser Parser Select col1,col2 From tab1 Where col3 > 5 TOK_QUERY TOK_FROM TOK_INSERT TOK_DESTINATION TOK_SELECT TOK_WHERE TOK_TABNAME TOK_SELEXPR TOK_SELEXPR tab1 TOK_DIR > TOK_TABLE_OR_COL TOK_TABLE_OR_COL col1 col2 QB TOK_TMP_FILE TOK_TABLE_OR_COL 5 insclause-0Friday, July 1, 2011

Parser Parser Select col1,col2 From tab1 Where col3 > 5 TOK_QUERY TOK_FROM TOK_INSERT TOK_DESTINATION TOK_SELECT TOK_WHERE QB TOK_TABNAME TOK_SELEXPR TOK_SELEXPR tab1 TOK_DIR > TOK_TABLE_OR_COL TOK_TABLE_OR_COL col1 col2 TOK_TMP_FILE TOK_TABLE_OR_COL 5 insclause-0Friday, July 1, 2011

Plan Plan Select col1,col2 From tab1 Where col3 > 5 QBFriday, July 1, 2011

Plan Plan Select col1,col2 From tab1 Where col3 > 5 QB TOK_FROM TOK_WHERE TOK_SELECT TOK_DESTINATIONFriday, July 1, 2011

Plan Plan Select col1,col2 From tab1 Where col3 > 5 QB TOK_FROM TableScanOperator TOK_WHERE TOK_SELECT TOK_DESTINATIONFriday, July 1, 2011

Plan Plan Select col1,col2 From tab1 Where col3 > 5 QB TOK_FROM TableScanOperator TOK_WHERE FilterOperator TOK_SELECT TOK_DESTINATIONFriday, July 1, 2011

Plan Plan Select col1,col2 From tab1 Where col3 > 5 QB TOK_FROM TableScanOperator TOK_WHERE FilterOperator TOK_SELECT SelectOperator TOK_DESTINATIONFriday, July 1, 2011

Plan Plan Select col1,col2 From tab1 Where col3 > 5 QB TOK_FROM TableScanOperator TOK_WHERE FilterOperator TOK_SELECT SelectOperator TOK_DESTINATION FileSinkOperatorFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 TableScanOperator FilterOperator SelectOperator FileSinkOperatorFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 tab1 {col1, col2, col3, col4,col5,col6,col7} TableScanOperator FilterOperator SelectOperator FileSinkOperatorFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 tab1 {col1, col2, col3, col4,col5,col6,col7} Context TableScanOperator FilterOperator ColumnPruner SelectOperator FileSinkOperatorFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 tab1 {col1, col2, col3, col4,col5,col6,col7} Context TableScanOperator FilterOperator FIL ColumnPruner TS SEL SelectOperator FileSinkOperatorFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 tab1 {col1, col2, col3, col4,col5,col6,col7} TableScanOperator FilterOperator FIL ColumnPruner TS SEL SelectOperator FileSinkOperator ContextFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 tab1 {col1, col2, col3, col4,col5,col6,col7} TableScanOperator FilterOperator ColumnPruner SelectOperator FIL FileSinkOperator Context TS SELFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 tab1 {col1, col2, col3, col4,col5,col6,col7} TableScanOperator FilterOperator ColumnPruner FIL SelectOperator Context TS SEL FileSinkOperatorFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 tab1 {col1, col2, col3, col4,col5,col6,col7} TableScanOperator FilterOperator ColumnPruner FIL SelectOperator Context TS SEL col1, col2 FileSinkOperatorFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 tab1 {col1, col2, col3, col4,col5,col6,col7} TableScanOperator FilterOperator ColumnPruner FIL SelectOperator Context TS SEL FileSinkOperatorFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 tab1 {col1, col2, col3, col4,col5,col6,col7} TableScanOperator FIL col1, col2, col3 FilterOperator Context TS ColumnPruner SEL SelectOperator FileSinkOperatorFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 tab1 {col1, col2, col3, col4,col5,col6,col7} TableScanOperator FIL FilterOperator Context TS ColumnPruner SEL SelectOperator FileSinkOperatorFriday, July 1, 2011

Optimizer Optimizer Select col1,col2 From tab1 Where col3 > 5 tab1 {col1, col2, col3, col4,col5,col6,col7} FIL TableScanOperator Context TS col1, col2, col3 SEL FilterOperator ColumnPruner FilterOperator SelectOperator FileSinkOperatorFriday, July 1, 2011

Task Task Select col1,col2 From tab1 Where col3 > 5 TS - GenMRTableScan1 TaskFactory FS - GenMRFileSink1 QBFriday, July 1, 2011

Task Task Select col1,col2 From tab1 Where col3 > 5 TS - GenMRTableScan1 TaskFactory FS - GenMRFileSink1 QB FetchTaskFriday, July 1, 2011

Task Task Select col1,col2 From tab1 Where col3 > 5 TS - GenMRTableScan1 TaskFactory FS - GenMRFileSink1 QB TableScanOperator FilterOperator FetchTask FilterOperator SelectOperator FileSinkOperatorFriday, July 1, 2011

Task Task Select col1,col2 From tab1 Where col3 > 5 TaskFactory FS - GenMRFileSink1 QB MapRedTask TableScanOperator FilterOperator FetchTask FilterOperator SelectOperator FileSinkOperatorFriday, July 1, 2011

Task Task Select col1,col2 From tab1 Where col3 > 5 TaskFactory QB MapRedTask TableScanOperator FilterOperator FetchTask FilterOperator SelectOperator FileSinkOperatorFriday, July 1, 2011

Task Task Select col1,col2 From tab1 Where col3 > 5 TaskFactory QB MapRedTask MapRedTask TableScanOperator FilterOperator FetchTask FilterOperator SelectOperator FileSinkOperatorFriday, July 1, 2011

Hive Internal Map Reduce Web UI Hive CLI JDBC TSOperator User Script Browse, Query, DDL UDF FILOperator SELOperator MetaStore Hive QL FILOperator FSOperator Thrift API Parser ExecMapper/ExecReducer Plan SerDe Optimizer Input/OutputFormat Task HDFS StorageHandler RCFile DB ... HBaseFriday, July 1, 2011

Oracle Migration to HiveFriday, July 1, 2011

l l l l Friday, July 1, 2011

l l l l l l l l Friday, July 1, 2011

Understand Oracle SQL • more than 3000 ETL SQL • understand Data-Flow • Group similar SQL Pattern • Investigate used Oracle FunctionFriday, July 1, 2011

Oracle SQLFriday, July 1, 2011

Data Model ConvertFriday, July 1, 2011

Data Model Convert TableFriday, July 1, 2011

Data Model Convert Table TableFriday, July 1, 2011

Data Model Convert Table Table PartitionFriday, July 1, 2011

Data Model Convert Table Table Partition PartitionFriday, July 1, 2011

Data Model Convert Table Table Partition Partition SamplingFriday, July 1, 2011

Data Model Convert Table Table Partition Partition Sampling BucketFriday, July 1, 2011

DataType ConvertFriday, July 1, 2011

DataType Convert NUMBER(n)Friday, July 1, 2011

DataType Convert NUMBER(n) TINYINT INT/BIGINTFriday, July 1, 2011

DataType Convert NUMBER(n) TINYINT INT/BIGINT NUMBER(n,m)Friday, July 1, 2011

DataType Convert NUMBER(n) TINYINT INT/BIGINT NUMBER(n,m) FLOAT/DOUBLEFriday, July 1, 2011

DataType Convert NUMBER(n) TINYINT INT/BIGINT NUMBER(n,m) FLOAT/DOUBLE VARCHAR2Friday, July 1, 2011

DataType Convert NUMBER(n) TINYINT INT/BIGINT NUMBER(n,m) FLOAT/DOUBLE VARCHAR2 STRINGFriday, July 1, 2011

DataType Convert NUMBER(n) TINYINT INT/BIGINT NUMBER(n,m) FLOAT/DOUBLE VARCHAR2 STRING DATEFriday, July 1, 2011

DataType Convert NUMBER(n) TINYINT INT/BIGINT NUMBER(n,m) FLOAT/DOUBLE VARCHAR2 STRING DATE STRING “yyyy-MM-dd HH:mm:ss” formatFriday, July 1, 2011

HIVE DML • HIVE supports ANSI-SQL • Only Support Sub-Queries in FROM clause • Join query : equi-join/inner-join outer-join self-joinFriday, July 1, 2011

IN ClauseFriday, July 1, 2011

IN Clause IN SubQueryFriday, July 1, 2011

IN Clause IN SubQuery SELECT * from Employee e WHERE e.DeptNo IN(SELECT d.DeptNo FROM Dept d)Friday, July 1, 2011

IN Clause IN SubQuery SELECT * from Employee e WHERE e.DeptNo IN(SELECT d.DeptNo FROM Dept d) SELECT * from Employee e LEFT SEMI JOIN Dept d ON (e.DeptNo=d.DeptNo)Friday, July 1, 2011

NOT IN ClauseFriday, July 1, 2011

NOT IN Clause NOT IN SubQueryFriday, July 1, 2011

NOT IN Clause NOT IN SubQuery SELECT * from Employee e WHERE e.DeptNo NOT IN(SELECT d.DeptNo FROM Dept d)Friday, July 1, 2011

NOT IN Clause NOT IN SubQuery SELECT * from Employee e WHERE e.DeptNo NOT IN(SELECT d.DeptNo FROM Dept d) SELECT e.* from Employee e LEFT OUTER JOIN Dept d ON (e.DeptNo=d.DeptNo) WHERE d.DeptNo IS NULLFriday, July 1, 2011

JOIN OperatorFriday, July 1, 2011

JOIN Operator JOINFriday, July 1, 2011

JOIN Operator JOIN SELECT * FROM Employee e1, Dept d1 WHERE e1.ID = d1.IdFriday, July 1, 2011

JOIN Operator JOIN SELECT * FROM Employee e1, Dept d1 WHERE e1.ID = d1.Id SELECT * FROM Employee e1 JOIN Dept d1 ON (e1.ID = d1.Id )Friday, July 1, 2011

Oracle FunctionFriday, July 1, 2011

FunctionsFriday, July 1, 2011

Functions Math Function round,ceil,mod, power,sqrt,sin/cosFriday, July 1, 2011

Functions Math Function Math Function round,ceil,mod, round,ceil,pmod, power,sqrt,sin/cos power,sqrt,sin/cosFriday, July 1, 2011

Functions Math Function Math Function round,ceil,mod, round,ceil,pmod, power,sqrt,sin/cos power,sqrt,sin/cos Character Function substr,trim,lpad/rpad ltrim/rtrim,replaceFriday, July 1, 2011

Functions Math Function Math Function round,ceil,mod, round,ceil,pmod, power,sqrt,sin/cos power,sqrt,sin/cos Character Function Character Function substr,trim,lpad/rpad substr,trim,lpad/rpad ltrim/rtrim,replace ltrim/rtrim,regexp_replaceFriday, July 1, 2011

Functions Math Function Math Function round,ceil,mod, round,ceil,pmod, power,sqrt,sin/cos power,sqrt,sin/cos Character Function Character Function substr,trim,lpad/rpad substr,trim,lpad/rpad ltrim/rtrim,replace ltrim/rtrim,regexp_replace NULL Function coalesce,nvl,nvl2Friday, July 1, 2011

Functions Math Function Math Function round,ceil,mod, round,ceil,pmod, power,sqrt,sin/cos power,sqrt,sin/cos Character Function Character Function substr,trim,lpad/rpad substr,trim,lpad/rpad ltrim/rtrim,replace ltrim/rtrim,regexp_replace NULL Function NULL Function coalesce,nvl,nvl2 coalesceFriday, July 1, 2011

Functions Math Function Math Function round,ceil,mod, round,ceil,pmod, power,sqrt,sin/cos power,sqrt,sin/cos Character Function Character Function substr,trim,lpad/rpad substr,trim,lpad/rpad ltrim/rtrim,replace ltrim/rtrim,regexp_replace NULL Function NULL Function coalesce,nvl,nvl2 coalesce No NVL,NVL2Friday, July 1, 2011

Custom UDF Function • Condition Function • DECODE, GREATEST • Null Comparison Function • NVL / NVL2 • Type Conversion • TO_NUMBER • TO_CHAR • TO_DATE • INSTR4 • DATE_FORMAT • LAST_DAYFriday, July 1, 2011

Oracle Analytic FunctionFriday, July 1, 2011

Analytic FunctionFriday, July 1, 2011

Analytic Function RANKFriday, July 1, 2011

Analytic Function RANK SELECT name,dept,salary,RANK() OVER (PARTITION BY dept ORDER BY salary DESC) FROM empFriday, July 1, 2011

Analytic Function RANK SELECT name,dept,salary,RANK() OVER (PARTITION BY dept ORDER BY salary DESC) FROM emp SELECT e.name,e.dept,e.salary,RANK( e.dept,e.salary) FROM (SELECT name, dept, salary FROM empDISTRIBUTED BY dept SORT BY dept, salary DESC) eFriday, July 1, 2011

Analytic Function RANK SELECT name,dept,salary,RANK() OVER (PARTITION BY dept ORDER BY salary DESC) FROM emp RANK(arg1,arg2) - Custom UDF SELECT e.name,e.dept,e.salary,RANK( e.dept,e.salary) FROM (SELECT name, dept, salary FROM empDISTRIBUTED BY dept SORT BY dept, salary DESC) eFriday, July 1, 2011

Analytic Aggregation FunctionFriday, July 1, 2011

Analytic Aggregation Function MINFriday, July 1, 2011

Analytic Aggregation Function MIN SELECT dept, MIN(salary) OVER (PARTITION BY dept) FROM empFriday, July 1, 2011

Analytic Aggregation Function MIN SELECT dept, MIN(salary) OVER (PARTITION BY dept) FROM emp SELECT dept,tmp.m FROM emp JOIN (SELECT dept, MIN(salary) m FROM emp GROUP BY dept) tmp ON emp.dept = tmp.deptFriday, July 1, 2011

Analytic Aggregation Function MIN SELECT dept, MIN(salary) OVER (PARTITION BY dept) FROM emp Aggregation + JOIN SELECT dept,tmp.m FROM emp JOIN (SELECT dept, MIN(salary) m FROM emp GROUP BY dept) tmp ON emp.dept = tmp.deptFriday, July 1, 2011

Hive InternalFriday, July 1, 2011

Merge Join Tree Bug • select * from a join b on a.v1 = b.v1 join c on a.v1 = c.v1 join d on a.v1 = d.v1 join e on a.v2 = e.v2 • select * from a join e on a.v2 = e.v2 join c on a.v1 = c.v1 join d on a.v1 = d.v1 join b on a.v1 = b.v1Friday, July 1, 2011

Merge Join Tree Bug • select * from a join b on a.v1 = b.v1 join c on a.v1 = c.v1 join d on a.v1 = d.v1 educ e #3 M a pR join e on a.v2 = e.v2 • select * from a join e on a.v2 = e.v2 join c on a.v1 = c.v1 join d on a.v1 = d.v1 join b on a.v1 = b.v1Friday, July 1, 2011

Merge Join Tree Bug • select * from a join b on a.v1 = b.v1 join c on a.v1 = c.v1 join d on a.v1 = d.v1 educ e #3 M a pR join e on a.v2 = e.v2 • select * from a join e on a.v2 = e.v2 join c on a.v1 = c.v1 join d on a.v1 = d.v1 duc e #2 Ma pRe join b on a.v1 = b.v1Friday, July 1, 2011

Merge Join Tree Bug Fix • SemanticAnalyzer private void mergeJoinTree(QB qb) { QBJoinTree root = qb.getQbJoinTree(); QBJoinTree parent = null; while (root != null) { boolean merged = mergeJoinNodes(qb, parent, root, root.getJoinSrc()); if (parent == null) { if (merged) { root = qb.getQbJoinTree(); } else { parent = root; root = root.getJoinSrc(); } } else { parent = parent.getJoinSrc(); root = parent.getJoinSrc(); }Friday, July 1, 2011

Merge Join Tree Bug Fix • SemanticAnalyzer private void mergeJoinTree(QB qb) { QBJoinTree root = qb.getQbJoinTree(); QBJoinTree parent = null; while (root != null) { boolean merged = mergeJoinNodes(qb, parent, root, root.getJoinSrc()); if (parent == null) { if (merged) { root = qb.getQbJoinTree(); } else { parent = root; root = root.getJoinSrc(); } } else { } else { if parent = parent.getJoinSrc(); (merged) { root = parent.getJoinSrc(); root = qb.getQbJoinTree(); } } else { parent = parent.getJoinSrc(); root = parent.getJoinSrc(); } }Friday, July 1, 2011

New HQL SyntaxFriday, July 1, 2011

New HQL Syntax INSERT INTOFriday, July 1, 2011

New HQL Syntax INSERT INTO INSERT INTO table VALUES(col1 ... coln) SELECT ... FROM tmp ...Friday, July 1, 2011

New HQL Syntax INSERT INTO INSERT INTO table VALUES(col1 ... coln) SELECT ... FROM tmp ... • INSERT [OVERWRITE] destination • grammar • modify FileSinkPlan • New Feature - HIVE-306 • INSERT INTO destinationFriday, July 1, 2011

TuningFriday, July 1, 2011

Tuning • Hadoop TunningFriday, July 1, 2011

Tuning • Hadoop Tunning • mapred.job.reuse.jvm.num.taskFriday, July 1, 2011

Tuning • Hadoop Tunning • mapred.job.reuse.jvm.num.task • mapred.child.java.optsFriday, July 1, 2011

Tuning • Hadoop Tunning • mapred.job.reuse.jvm.num.task • mapred.child.java.opts • mapred.min.split.size / mapred.max.split.sizeFriday, July 1, 2011

Tuning • Hadoop Tunning • mapred.job.reuse.jvm.num.task • mapred.child.java.opts • mapred.min.split.size / mapred.max.split.size • dfs.block.sizeFriday, July 1, 2011

Tuning • Hadoop Tunning • mapred.job.reuse.jvm.num.task • mapred.child.java.opts • mapred.min.split.size / mapred.max.split.size • dfs.block.size • Hive TunningFriday, July 1, 2011

Tuning • Hadoop Tunning • mapred.job.reuse.jvm.num.task • mapred.child.java.opts • mapred.min.split.size / mapred.max.split.size • dfs.block.size • Hive Tunning • hive.input.format = CombineHiveInputFormatFriday, July 1, 2011

Tuning • Hadoop Tunning • mapred.job.reuse.jvm.num.task • mapred.child.java.opts • mapred.min.split.size / mapred.max.split.size • dfs.block.size • Hive Tunning • hive.input.format = CombineHiveInputFormat • query tuning - reduce # of MapReduce using HQL PlanFriday, July 1, 2011

Wrap-Up Oracle 2 HiveFriday, July 1, 2011

Wrap-Up Oracle 2 Hive Look insight data ﬂow & modelFriday, July 1, 2011

Wrap-Up Oracle 2 Hive Look insight data ﬂow & model Modify Oracle SQL to Hive Query SyntaxFriday, July 1, 2011

Wrap-Up Oracle 2 Hive Look insight data ﬂow & model Modify Oracle SQL to Hive Query Syntax Use Built-in functionFriday, July 1, 2011

Wrap-Up Oracle 2 Hive Look insight data ﬂow & model Modify Oracle SQL to Hive Query Syntax Use Built-in function Develop custom UDF/UDAF/UDTFFriday, July 1, 2011

Wrap-Up Oracle 2 Hive Look insight data ﬂow & model Modify Oracle SQL to Hive Query Syntax Use Built-in function Develop custom UDF/UDAF/UDTF Support analytic functionFriday, July 1, 2011

Wrap-Up Oracle 2 Hive Look insight data ﬂow & model Modify Oracle SQL to Hive Query Syntax Use Built-in function Develop custom UDF/UDAF/UDTF Support analytic function - distributed by + sort by + udfFriday, July 1, 2011

Wrap-Up Oracle 2 Hive Look insight data ﬂow & model Modify Oracle SQL to Hive Query Syntax Use Built-in function Develop custom UDF/UDAF/UDTF Support analytic function - distributed by + sort by + udf - join + udf (aggregation)Friday, July 1, 2011

Wrap-Up Oracle 2 Hive Look insight data ﬂow & model Modify Oracle SQL to Hive Query Syntax Use Built-in function Develop custom UDF/UDAF/UDTF Support analytic function - distributed by + sort by + udf - join + udf (aggregation) Modify internal hiveFriday, July 1, 2011

Wrap-Up Oracle 2 Hive Look insight data ﬂow & model Modify Oracle SQL to Hive Query Syntax Use Built-in function Develop custom UDF/UDAF/UDTF Support analytic function - distributed by + sort by + udf - join + udf (aggregation) Modify internal hive Hadoop + Hive TunningFriday, July 1, 2011

Question ?Friday, July 1, 2011

http://d.hatena.ne.jp	596
http://translate.googleusercontent.com	12
http://twitter.com	4
http://webcache.googleusercontent.com	3
http://paper.li	1
http://b.hatena.ne.jp	1

Replacing Telco DB/DW to Hadoop and Hive

by JunHo Cho on Jul 01, 2011

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 617

Statistics

Replacing Telco DB/DW to Hadoop and Hive Presentation Transcript