Gruter, profile picture
Uploaded byGruter
4,817 views

Introduction to Apache Tajo: Data Warehouse for Big Data

Apache Tajo is a top-level Apache project functioning as a big data warehouse, enabling ANSI-SQL compliant querying across diverse storage types while ensuring fault tolerance and efficient execution. The recent release highlights features like query federation, JDBC-based storage support, and self-describing data formats, enhancing performance and user experience. Tajo finds applications in various sectors, providing solutions for data analysis, ETL workloads, and ad-hoc queries, significantly reducing operational costs.

Embed presentation

Introduction to Apache Tajo: Data Warehouse for Big Data Jihoon Son / Gruter inc.
About Me Jihoon Son (@jihoonson) Tajo project co-founder Committer and PMC member of Apache Tajo Research engineer at Gruter 2
Outline About Tajo Features of the Recent Release Demo Roadmap 3
What is Tajo? Tajo / tάːzo / 타조 An ostrich in Korean The world's fastest two-legged animal 4
What is Tajo? Apache Top-level Project Big data warehouse system ANSI-SQL compliant Mature SQL features Various types of join, window functions Rapid query execution with own distributed DAG engine Low latency, and long running batch queries with a single system Fault-tolerance Beyond SQL-on-Hadoop Support various types of storage 5
Tajo Master Catalog Server Tajo Master Catalog Server Architecture Overview 6 DBMS HCatalog Tajo Master Catalog Server Tajo Worker Query Master Query Executor Storage Service Tajo Worker Query Master Query Executor Storage Service Tajo Worker Query Master Query Executor Storage Service JDBC client TSQLWebUI REST API Storage Submit a query Manage metadataAllocate a query Send tasks & monitor Send tasks & monitor
Who are Using Tajo? Use cases: replacement of commercial DW 1st telco in South Korea Replacement of long-running ETL workloads on several TB datasets Lots of daily reports about user behavior Ad-‐hoc analysis on TB datasets Benefits Simplified architecture for data analysis An unified system for DW ETL, OLAP, and Hadoop ETL Much less cost, more data analysis within same SLA Saved license fee of commercial DW 7
Who are Using Tajo? Use cases: data discovery Music streaming service (26 million users) Analysis of purchase history for target marketing Benefits Interactive query on large datasets Data analysis with familiar BI tools 8
Recent Release: 0.11 Feature highlights Query federation JDBC-based storage support Self-describing data formats support Multi-query support More stable and efficient join execution Index support Python UDF/UDAF support 9
Recent Release: 0.11 Today's topic Query federation JDBC-based storage support Self-describing data formats support Multi-query support More stable and efficient join execution Index support Python UDF/UDAF support 10
Query Federation with Tajo 11
Your Data Your data might be spread on multiple heterogeneous sites Cloud, DBMS, Hadoop, NoSQL, … 12 DBMS Application Cloud storage On-premise storage NoSQL
Your Data Even in a single site, your data might be stored in different data formats 13 JSONCSV Parque t ORC Log ...
Your Data How to analyze distributed data? Traditionally ... 14 DBMSApplication Cloud storage On-premise storage NoSQL Global view ETL transform ● Long delivery ● Complex data flow ● Human-intensive
Your Data with Tajo Query federation 15 DBMSApplication Cloud storage On-premise storage NoSQL Global view ● Fast delivery ● Easy maintenance ● Simple data flow
Storage and Data Format Support 16 Data formats Storage types
Create Table > CREATE EXTERNAL TABLE archive1 (id BIGINT, ...) USING text WITH ('text.delimiter'='|') LOCATION 'hdfs://localhost:8020/archive1'; > CREATE EXTERNAL TABLE user (user_id BIGINT, ...) USING orc WITH ('orc.compression.kind'='snappy') LOCATION 's3://user'; > CREATE EXTERNAL TABLE table1 (key TEXT, ...) USING hbase LOCATION 'hbase:zk://localhost:2181/uptodate'; > ... 17 Data formatStorage URI
Create Table > CREATE EXTERNAL TABLE archive1 (id BIGINT, ...) USING text WITH ('text.delimiter'='|','text.null'='N','compression.codec'='org.apache.hadoop.io.compr ess.SnappyCodec','timezone'='UTC+9','text.skip.headerlines'='2') LOCATION 'hdfs://localhost:8020/tajo/warehouse/archive1'; > CREATE EXTERNAL TABLE archive2 (id BIGINT, ...) USING text WITH ('text.delimiter'='|','text.null'='N','compression.codec'='org.apache.hadoop.io.compr ess.SnappyCodec','timezone'='UTC+9','text.skip.headerlines'='2') LOCATION 'hdfs://localhost:8020/tajo/warehouse/archive2'; > CREATE EXTERNAL TABLE archive3 (id BIGINT, ...) USING text WITH ('text.delimiter'='|','text.null'='N','compression.codec'='org.apache.hadoop.io.compr ess.SnappyCodec','timezone'='UTC+9','text.skip.headerlines'='2') LOCATION 'hdfs://localhost:8020/tajo/warehouse/archive3'; > ... 18 Too tedious!
Introduction to Tablespace Tablespace Registered storage space A tablespace is identified by an unique URI Configurations and policies are shared by all tables in a tablespace Storage type Default data format and supported data formats It allows users to reuse registered storage configurations and policies 19
Tablespaces, Databases, and Tables 20 Namespace Storage1 Storage2 ... ... ... Tablespace1 Tablespace2 Tablespace3 Physical space Table1 Table2 Table3 Database1 Database1 ...
Tablespace Configuration { "spaces" : { "warehouse" : { "uri" : "hdfs://localhost:8020/tajo/warehouse", "configs" : [ {'text.delimiter'='|'}, {'text.null'='N'}, {'compression.codec'='org.apache.hadoop.io.compress.SnappyCodec'}, {'timezone'='UTC+9'}, {'text.skip.headerlines'='2'} ] }, "hbase1" : { "uri" : "hbase:zk://localhost:2181/table1" } } } 21 Tablespace name Tablespace URI
Create Table > CREATE TABLE archive1 (id BIGINT, ...) TABLESPACE warehouse; 22 Tablespace nameData format is omitted. Default data format is TEXT. "warehouse" : { "uri" : "hdfs://localhost:8020/tajo/warehouse", "configs" : [ {'text.delimiter'='|'}, {'text.null'='N'}, {'compression.codec'='org.apache.hadoop.io.compress.SnappyCodec'}, {'timezone'='UTC+9'}, {'text.skip.headerlines'='2'} ] },
Create Table > CREATE TABLE archive1 (id BIGINT, ...) TABLESPACE warehouse; > CREATE TABLE archive2 (id BIGINT, ...) TABLESPACE warehouse; > CREATE TABLE archive3 (id BIGINT, ...) TABLESPACE warehouse; > CREATE TABLE user (user_id BIGINT, ...) TABLESPACE aws USING orc WITH ('orc.compression.kind'='snappy'); > CREATE TABLE table1 (key TEXT, ...) TABLESPACE hbase1; > ... 23
HDFS HBase Tajo Worker Query Engine Storage ServiceHDFS handler Tajo Worker Query Engine Storage ServiceHDFS handler Tajo Worker Query Engine Storage ServiceHBase handler Querying on Different Data Silos How does a worker access different data sources? Storage service Return a proper handler for underlying storage > SELECT ... FROM hdfs_table, hbase_table, ... 24
JDBC-based Storage Support 25
jdbc_db1 tajo_db1 JDBC-based Storage Storage providing the JDBC interface PostgreSQL, MySQL, MariaDB, ... Databases of JDBC-based storage are mapped to Tajo databases 26 Table 1 Table 2 Table 3 Table 1 Table 2 Table 3 tajo_db2 Table 1 Table 2 Table 3 … jdbc_db2 Table 1 Table 2 Table 3 … JDBC-based storage Tajo
Tablespace Configuration { "spaces": { "pgsql_db1": { "uri": "jdbc:postgresql://hostname:port/db1" "configs": { "mapped_database": "tajo_db1" "connection_properties": { "user": "tajo", "password": "xxxx" } } } } } 27 PostgreSQL database name Tajo database name Tablespace name
Return to Query Federation How to correlate data on JDBC-based storage and others? Need to have a global view of metadata across different storage types Tajo also has its own metadata for its data Each JDBC-based storage has own metadata for its data Each NoSQL storage has metadata for its data … 28
Metadata Federation Federating metadata of underlying storage 29 DBMS metadata provider NoSQL metadata provider Linked Metadata Manager DBMS HCatalog Tajo catalog metadata provider Catalog Interface ● Tablespace ● Database ● Tables ● Schema names ...
Querying on JDBC-based Storage A plan is converted into a SQL string Query generation Diverse SQL syntax of different types of storage Different SQL builder for each storage type 30 Tajo Master Tajo Worker JDBC-based storage SELECT ... Query plan SELECT ...
Operation Push Down Tajo can exploit the processing capability of underlying storage DBMSs, MongoDB, HBase, … Operations are pushed down into underlying storage Leveraging the advanced features provided by underlying storage Ex) DBMSs' query optimization, index, ... 31
Example 1 32 SELECT count(*) FROM account ac, archive ar WHERE ac.key = ar.id and ac.name = 'tajo' account DBMS archive HDFS scan archive scan account ac.name = 'tajo' join ac.key = ar.id group by count(*) group by count(*) Full scan Result only Push operation
Example 2 33 SELECT ac.name, count(*) FROM account ac GROUP BY ac.name account DBMS scan account group by count(*) Result only Push operation
Self-describing Data Formats Support 34
Self-describing Data Formats Some data formats include schema information as well as data JSON, ORC, Parquet, … Tajo 0.11 natively supports self-describing data formats Since they already have schema information, Tajo doesn't need to store it aside Instead, Tajo can infer the schema at query execution time 35
Create Table with Nested Data Format { "title" : "Hand of the King", "name" : { "first_name": "Eddard", "last_name": "Stark"}} { "title" : "Assassin", "name" : { "first_name": "Arya", "last_name": "Stark"}} { "title" : "Dancing Master", "name" : { "first_name": "Syrio", "last_name": "Forel"}} > CREATE EXTERNAL TABLE schemaful_table ( title TEXT, name RECORD ( first_name TEXT, last_name TEXT ) ) USING json LOCATION 'hdfs:///json_table'; 36 Nested type
How about This Data? {"id":"2937257761","type":"ForkEvent","actor":{"id":1088854,"login":"CAOakleyII","gravatar_id":"","url":"https://api.github.com/users/CAOakleyII","avatar_url":"https://avatars.githubusercontent.com/ u/1088854?"},"repo":{"id":11909954,"name":"skycocker/chromebrew","url":"https://api.github.com/repos/skycocker/chromebrew"},"payload":{"forkee":{"id":38339291,"name":"chromebrew","full_na me":"CAOakleyII/chromebrew","owner":{"login":"CAOakleyII","id":1088854,"avatar_url":"https://avatars.githubusercontent.com/u/1088854?v=3","gravatar_id":"","url":"https://api.github.com/users/C AOakleyII","html_url":"https://github.com/CAOakleyII","followers_url":"https://api.github.com/users/CAOakleyII/followers","following_url":"https://api.github.com/users/CAOakleyII/following{/other_ user}","gists_url":"https://api.github.com/users/CAOakleyII/gists{/gist_id}","starred_url":"https://api.github.com/users/CAOakleyII/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/ users/CAOakleyII/subscriptions","organizations_url":"https://api.github.com/users/CAOakleyII/orgs","repos_url":"https://api.github.com/users/CAOakleyII/repos","events_url":"https://api.github.com/ users/CAOakleyII/events{/privacy}","received_events_url":"https://api.github.com/users/CAOakleyII/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/C AOakleyII/chromebrew","description":"Package manager for Chrome OS","fork":true,"url":"https://api.github.com/repos/CAOakleyII/chromebrew","forks_url":"https://api.github.com/repos/CAOakleyII/chromebrew/forks","keys_url":"https://api.github.com/repos/CAOa kleyII/chromebrew/keys{/key_id}","collaborators_url":"https://api.github.com/repos/CAOakleyII/chromebrew/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/CAOakleyII/chro mebrew/teams","hooks_url":"https://api.github.com/repos/CAOakleyII/chromebrew/hooks","issue_events_url":"https://api.github.com/repos/CAOakleyII/chromebrew/issues/events{/number}","event s_url":"https://api.github.com/repos/CAOakleyII/chromebrew/events","assignees_url":"https://api.github.com/repos/CAOakleyII/chromebrew/assignees{/user}","branches_url":"https://api.github.com /repos/CAOakleyII/chromebrew/branches{/branch}","tags_url":"https://api.github.com/repos/CAOakleyII/chromebrew/tags","blobs_url":"https://api.github.com/repos/CAOakleyII/chromebrew/git/blo bs{/sha}","git_tags_url":"https://api.github.com/repos/CAOakleyII/chromebrew/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/CAOakleyII/chromebrew/git/refs{/sha}","trees_url":"https:/ /api.github.com/repos/CAOakleyII/chromebrew/git/trees{/sha}","statuses_url":"https://api.github.com/repos/CAOakleyII/chromebrew/statuses/{sha}","languages_url":"https://api.github.com/repos/C AOakleyII/chromebrew/languages","stargazers_url":"https://api.github.com/repos/CAOakleyII/chromebrew/stargazers","contributors_url":"https://api.github.com/repos/CAOakleyII/chromebrew/cont ributors","subscribers_url":"https://api.github.com/repos/CAOakleyII/chromebrew/subscribers","subscription_url":"https://api.github.com/repos/CAOakleyII/chromebrew/subscription","commits_url": "https://api.github.com/repos/CAOakleyII/chromebrew/commits{/sha}","git_commits_url":"https://api.github.com/repos/CAOakleyII/chromebrew/git/commits{/sha}","comments_url":"https://api.gith ub.com/repos/CAOakleyII/chromebrew/comments{/number}","issue_comment_url":"https://api.github.com/repos/CAOakleyII/chromebrew/issues/comments{/number}","contents_url":"https://api.git hub.com/repos/CAOakleyII/chromebrew/contents/{+path}","compare_url":"https://api.github.com/repos/CAOakleyII/chromebrew/compare/{base}...{head}","merges_url":"https://api.github.com/repo s/CAOakleyII/chromebrew/merges","archive_url":"https://api.github.com/repos/CAOakleyII/chromebrew/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/CAOakleyII/chromebr ew/downloads","issues_url":"https://api.github.com/repos/CAOakleyII/chromebrew/issues{/number}","pulls_url":"https://api.github.com/repos/CAOakleyII/chromebrew/pulls{/number}","milestones_ url":"https://api.github.com/repos/CAOakleyII/chromebrew/milestones{/number}","notifications_url":"https://api.github.com/repos/CAOakleyII/chromebrew/notifications{?since,all,participating}","lab els_url":"https://api.github.com/repos/CAOakleyII/chromebrew/labels{/name}","releases_url":"https://api.github.com/repos/CAOakleyII/chromebrew/releases{/id}","created_at":"2015-07- 01T00:00:00Z","updated_at":"2015-06-28T10:11:09Z","pushed_at":"2015-06- 09T07:46:57Z","git_url":"git://github.com/CAOakleyII/chromebrew.git","ssh_url":"git@github.com:CAOakleyII/chromebrew.git","clone_url":"https://github.com/CAOakleyII/chromebrew.git","svn_url":" https://github.com/CAOakleyII/chromebrew","homepage":"http://skycocker.github.io/chromebrew/","size":846,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":false,"has_downl oads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master","public":true}},"public":true,"cre ated_at":"2015-07-01T00:00:01Z"} ... 37
Create Schemaless Table > CREATE EXTERNAL TABLE schemaless_table (*) USING json LOCATION 'hdfs:///json_table'; That's all! 38 Allow any schema
Schema-free Query Execution > CREATE EXTERNAL TABLE schemaful_table (id BIGINT, name TEXT, ...) USING text LOCATION 'hdfs:///csv_table; > CREATE EXTERNAL TABLE schemaless_table (*) USING json LOCATION 'hdfs:///json_table'; > SELECT name.first_name, name.last_name from schemaless_table; > SELECT title, count(*) FROM schemaful_table, schemaless_table WHERE name = name.last_name GROUP BY title; 39
Schema Inference Table schema is inferred at query time Example 40 SELECT a, b.b1, b.b2.c1 FROM t; ( a text, b record ( b1 text, b2 record ( c1 text ) ) ) Inferred schemaQuery
Demo 41
Demo with Command line 42
Roadmap 43
Roadmap 0.12 Improved Yarn integration Authentication support JavaScript stored procedure support Scalar subquery support Hive UDF support 44
Roadmap Next generation (beyond 0.12) Exploiting modern hardware Approximate query processing Genetic query optimization And more … 45
tajo> select question from you; 46

More Related Content

PDF
Gruter TECHDAY 2014 Realtime Processing in Telco
byGruter
 
PPTX
Big Data Camp LA 2014 - Apache Tajo: A Big Data Warehouse System on Hadoop
byGruter
 
PPTX
Tajo Seoul Meetup July 2015 - What's New Tajo 0.11
byHyunsik Choi
 
PDF
What's New Tajo 0.10 and Its Beyond
byGruter
 
PDF
Ingesting hdfs intosolrusingsparktrimmed
bywhoschek
 
PPTX
Gruter_TECHDAY_2014_03_ApacheTajo (in Korean)
byGruter
 
PDF
Introduction to Apache Tajo: Future of Data Warehouse
byJihoon Son
 
PDF
Apache Tajo - Bay Area HUG Nov. 2013 LinkedIn Special Event
byGruter
 
Gruter TECHDAY 2014 Realtime Processing in Telco
byGruter
 
Big Data Camp LA 2014 - Apache Tajo: A Big Data Warehouse System on Hadoop
byGruter
 
Tajo Seoul Meetup July 2015 - What's New Tajo 0.11
byHyunsik Choi
 
What's New Tajo 0.10 and Its Beyond
byGruter
 
Ingesting hdfs intosolrusingsparktrimmed
bywhoschek
 
Gruter_TECHDAY_2014_03_ApacheTajo (in Korean)
byGruter
 
Introduction to Apache Tajo: Future of Data Warehouse
byJihoon Son
 
Apache Tajo - Bay Area HUG Nov. 2013 LinkedIn Special Event
byGruter
 

What's hot

PDF
[db tech showcase Tokyo 2017] C23: Lessons from SQLite4 by SQLite.org - Richa...
byInsight Technology, Inc.
 
PPTX
Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016
byDataStax
 
PDF
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
byDatabricks
 
PDF
User Defined Partitioning on PlazmaDB
byKai Sasaki
 
PDF
Vitalii Bondarenko HDinsight: spark. advanced in memory big-data analytics wi...
byАліна Шепшелей
 
PDF
Presto At Treasure Data
byTaro L. Saito
 
PPTX
Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB
byAthiq Ahamed
 
PPTX
Big data, just an introduction to Hadoop and Scripting Languages
byCorley S.r.l.
 
PPTX
Apache Kite
byAlwin James
 
PPTX
Tez Shuffle Handler: Shuffling at Scale with Apache Hadoop
byDataWorks Summit
 
PDF
From docker to kubernetes: running Apache Hadoop in a cloud native way
byDataWorks Summit
 
PDF
Introduction to Apache Tajo: Data Warehouse for Big Data
byJihoon Son
 
PDF
Efficient in situ processing of various storage types on apache tajo
byHyunsik Choi
 
PPTX
Hadoop Summit 2014: Query Optimization and JIT-based Vectorized Execution in ...
byGruter
 
PDF
Apache Tajo on Swift: Bringing SQL to the OpenStack World
byJihoon Son
 
PPT
July 2010 Triangle Hadoop Users Group - Chad Vawter Slides
byryancox
 
PDF
Introduction to Hadoop
byOvidiu Dimulescu
 
PDF
Tajo: A Distributed Data Warehouse System for Hadoop
byHyunsik Choi
 
[db tech showcase Tokyo 2017] C23: Lessons from SQLite4 by SQLite.org - Richa...
byInsight Technology, Inc.
 
Myths of Big Partitions (Robert Stupp, DataStax) | Cassandra Summit 2016
byDataStax
 
Hudi: Large-Scale, Near Real-Time Pipelines at Uber with Nishith Agarwal and ...
byDatabricks
 
User Defined Partitioning on PlazmaDB
byKai Sasaki
 
Vitalii Bondarenko HDinsight: spark. advanced in memory big-data analytics wi...
byАліна Шепшелей
 
Presto At Treasure Data
byTaro L. Saito
 
Benchmarking Top NoSQL Databases: Apache Cassandra, Apache HBase and MongoDB
byAthiq Ahamed
 
Big data, just an introduction to Hadoop and Scripting Languages
byCorley S.r.l.
 
Apache Kite
byAlwin James
 
Tez Shuffle Handler: Shuffling at Scale with Apache Hadoop
byDataWorks Summit
 
From docker to kubernetes: running Apache Hadoop in a cloud native way
byDataWorks Summit
 
Introduction to Apache Tajo: Data Warehouse for Big Data
byJihoon Son
 
Efficient in situ processing of various storage types on apache tajo
byHyunsik Choi
 
Hadoop Summit 2014: Query Optimization and JIT-based Vectorized Execution in ...
byGruter
 
Apache Tajo on Swift: Bringing SQL to the OpenStack World
byJihoon Son
 
July 2010 Triangle Hadoop Users Group - Chad Vawter Slides
byryancox
 
Introduction to Hadoop
byOvidiu Dimulescu
 
Tajo: A Distributed Data Warehouse System for Hadoop
byHyunsik Choi
 

Viewers also liked

PPTX
Introduction to Apache Tajo
byGruter
 
PPTX
Hadoop and Enterprise Data Warehouse
byDataWorks Summit
 
PPTX
How One Company Offloaded Data Warehouse ETL To Hadoop and Saved $30 Million
byDataWorks Summit
 
PPTX
Big Data Warehousing Meetup: Dimensional Modeling Still Matters!!!
byCaserta
 
PDF
MelOn 빅데이터 플랫폼과 Tajo 이야기
byGruter
 
PDF
Big data analysis with R and Apache Tajo (in Korean)
byGruter
 
PDF
Expanding Your Data Warehouse with Tajo
byGruter
 
PPTX
Elastic Search Performance Optimization - Deview 2014
byGruter
 
PDF
Introduction to Apache Tajo: Future of Data Warehouse
byGruter
 
PPTX
Best Practices for the Hadoop Data Warehouse: EDW 101 for Hadoop Professionals
byCloudera, Inc.
 
PDF
Hadoop Integration into Data Warehousing Architectures
byHumza Naseer
 
Introduction to Apache Tajo
byGruter
 
Hadoop and Enterprise Data Warehouse
byDataWorks Summit
 
How One Company Offloaded Data Warehouse ETL To Hadoop and Saved $30 Million
byDataWorks Summit
 
Big Data Warehousing Meetup: Dimensional Modeling Still Matters!!!
byCaserta
 
MelOn 빅데이터 플랫폼과 Tajo 이야기
byGruter
 
Big data analysis with R and Apache Tajo (in Korean)
byGruter
 
Expanding Your Data Warehouse with Tajo
byGruter
 
Elastic Search Performance Optimization - Deview 2014
byGruter
 
Introduction to Apache Tajo: Future of Data Warehouse
byGruter
 
Best Practices for the Hadoop Data Warehouse: EDW 101 for Hadoop Professionals
byCloudera, Inc.
 
Hadoop Integration into Data Warehousing Architectures
byHumza Naseer
 

Similar to Introduction to Apache Tajo: Data Warehouse for Big Data

PDF
"Building Data Warehouse with Google Cloud Platform", Artem Nikulchenko
byFwdays
 
PDF
Scale Relational Database with NewSQL
byPingCAP
 
PDF
TiDB for Big Data
byPingCAP
 
PDF
Apache TAJO
byAsis Mohanty
 
PDF
Big Data Day LA 2015 - What's New Tajo 0.10 and Beyond by Hyunsik Choi of Gruter
byData Con LA
 
PDF
Apache Tajo - An open source big data warehouse
byhadoopsphere
 
PPTX
Efficient In-situ Processing of Various Storage Types on Apache Tajo
byDataWorks Summit
 
PDF
Efficient In­‐situ Processing of Various Storage Types on Apache Tajo
byGruter
 
PDF
Building Operational Data Lake using Spark and SequoiaDB with Yang Peng
byDatabricks
 
PPTX
[FOSS4G 2015 SEOUL] Spatial tajo supporting spatial queries on Apache Tajo
byBD
 
PDF
SQL on Hadoop in Taiwan
byTreasure Data, Inc.
 
PDF
Treasure Data and OSS
byN Masahiro
 
PDF
Owning time series with team apache Strata San Jose 2015
byPatrick McFadin
 
PDF
Tajo_Meetup_20141120
byHyoungjun Kim
 
PPTX
Apache Tajo - BWC 2014
byGruter
 
PDF
In Memory Data Pipeline And Warehouse At Scale - BerlinBuzzwords 2015
byIulia Emanuela Iancuta
 
PDF
Interactive SQL-on-Hadoop and JethroData
byOfir Manor
 
PDF
Tajolabigdatacamp2014 140618135810-phpapp01 hyunsik-choi
byData Con LA
 
PPTX
Chen li asterix db： 大数据处理开源平台
byjins0618
 
PDF
Query optimization in Apache Tajo
byJihoon Son
 
"Building Data Warehouse with Google Cloud Platform", Artem Nikulchenko
byFwdays
 
Scale Relational Database with NewSQL
byPingCAP
 
TiDB for Big Data
byPingCAP
 
Apache TAJO
byAsis Mohanty
 
Big Data Day LA 2015 - What's New Tajo 0.10 and Beyond by Hyunsik Choi of Gruter
byData Con LA
 
Apache Tajo - An open source big data warehouse
byhadoopsphere
 
Efficient In-situ Processing of Various Storage Types on Apache Tajo
byDataWorks Summit
 
Efficient In­‐situ Processing of Various Storage Types on Apache Tajo
byGruter
 
Building Operational Data Lake using Spark and SequoiaDB with Yang Peng
byDatabricks
 
[FOSS4G 2015 SEOUL] Spatial tajo supporting spatial queries on Apache Tajo
byBD
 
SQL on Hadoop in Taiwan
byTreasure Data, Inc.
 
Treasure Data and OSS
byN Masahiro
 
Owning time series with team apache Strata San Jose 2015
byPatrick McFadin
 
Tajo_Meetup_20141120
byHyoungjun Kim
 
Apache Tajo - BWC 2014
byGruter
 
In Memory Data Pipeline And Warehouse At Scale - BerlinBuzzwords 2015
byIulia Emanuela Iancuta
 
Interactive SQL-on-Hadoop and JethroData
byOfir Manor
 
Tajolabigdatacamp2014 140618135810-phpapp01 hyunsik-choi
byData Con LA
 
Chen li asterix db： 大数据处理开源平台
byjins0618
 
Query optimization in Apache Tajo
byJihoon Son
 

More from Gruter

PDF
스타트업사례로 본 로그 데이터분석 : Tajo on AWS
byGruter
 
PPTX
Vectorized processing in_a_nutshell_DeView2014
byGruter
 
PPTX
DeView2013 Big Data Platform Architecture with Hadoop - Hyeong-jun Kim
byGruter
 
PDF
Tajo TPC-H Benchmark Test on AWS
byGruter
 
PPTX
Gruter TECHDAY 2014 MelOn BigData
byGruter
 
PDF
Cloumon sw제품설명회 발표자료
byGruter
 
PPTX
Tajo and SQL-on-Hadoop in Tech Planet 2013
byGruter
 
PPTX
SQL-on-Hadoop with Apache Tajo, and application case of SK Telecom
byGruter
 
PPTX
Hadoop security DeView 2014
byGruter
 
PDF
GRUTER가 들려주는 Big Data Platform 구축 전략과 적용 사례: Tajo와 SQL-on-Hadoop
byGruter
 
PDF
Tajo case study bay area hug 20131105
byGruter
 
PDF
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
byGruter
 
PDF
Data analysis with Tajo
byGruter
 
PDF
GRUTER가 들려주는 Big Data Platform 구축 전략과 적용 사례: Bioinformatics Data를 위한 Hadoop기반...
byGruter
 
PDF
GRUTER가 들려주는 Big Data Platform 구축 전략과 적용 사례: 온라인 컨텐츠 서비스를 위한 빅데이터 구축 사례
byGruter
 
PDF
Gruter_TECHDAY_2014_01_SearchEngine (in Korean)
byGruter
 
스타트업사례로 본 로그 데이터분석 : Tajo on AWS
byGruter
 
Vectorized processing in_a_nutshell_DeView2014
byGruter
 
DeView2013 Big Data Platform Architecture with Hadoop - Hyeong-jun Kim
byGruter
 
Tajo TPC-H Benchmark Test on AWS
byGruter
 
Gruter TECHDAY 2014 MelOn BigData
byGruter
 
Cloumon sw제품설명회 발표자료
byGruter
 
Tajo and SQL-on-Hadoop in Tech Planet 2013
byGruter
 
SQL-on-Hadoop with Apache Tajo, and application case of SK Telecom
byGruter
 
Hadoop security DeView 2014
byGruter
 
GRUTER가 들려주는 Big Data Platform 구축 전략과 적용 사례: Tajo와 SQL-on-Hadoop
byGruter
 
Tajo case study bay area hug 20131105
byGruter
 
Gruter_TECHDAY_2014_04_TajoCloudHandsOn (in Korean)
byGruter
 
Data analysis with Tajo
byGruter
 
GRUTER가 들려주는 Big Data Platform 구축 전략과 적용 사례: Bioinformatics Data를 위한 Hadoop기반...
byGruter
 
GRUTER가 들려주는 Big Data Platform 구축 전략과 적용 사례: 온라인 컨텐츠 서비스를 위한 빅데이터 구축 사례
byGruter
 
Gruter_TECHDAY_2014_01_SearchEngine (in Korean)
byGruter
 

Recently uploaded

PDF
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
PPTX
Presentation 1-PUM Singapore 2015- 2D.pptx
byWilliamChong71
 
PPT
Chapter 3 Reusable Technoldsadadsdsdasdogy.ppt
bytuananle9897
 
PPTX
software define network authentication .pptx
byAhmedGouda93
 
PDF
BCS501 Module-05 Software Engineering and Project Management.pdf
byjaadu6918
 
PPTX
[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...
byDataScienceConferenc1
 
PPTX
Copy of First ENGR 100 Presentation.pptx
bygenshinds971
 
PDF
Natal SM 2025 - Latihan Liturgi Kelas 0 - 3 SD.pdf
byWenny63
 
PPTX
[DSC Europe 25] Andy Cotgreave - Nothing is new in analytics.pptx
byDataScienceConferenc1
 
PPTX
Time serise forcasting of non-durble consumer goods industry
byvkumaranand89
 
PPTX
[DSC Europe 25] Vid Stimac - Policy Parsimony: Between Oversimplifying and Ov...
byDataScienceConferenc1
 
PPTX
Safe_Spaces_Act_Modern_Presentation.pptx
byairesmijares2003
 
PPTX
[DSC Europe 25] Dragana Ilic - AI for Big Data in Astronomy.pptx
byDataScienceConferenc1
 
PPTX
[DSC Europe 25] Jim Sterne - Adopting Generative AI Capabilities Into the Ent...
byDataScienceConferenc1
 
PPTX
Customer Segmentation: Seeing the Trees and the Forest Simultaneously - V2
bySK Palu
 
PDF
20251202 Frank_Melis Vittal_Prabhu Platform_Talk v3b.pdf
byhome
 
PPTX
1. Prepare for the application of health survey.pptx
byDegaregeNibret
 
PPTX
hindustan Unilever Limited Presentation - Sample
bymohitsawale3
 
PDF
Bicycle Project sostenible pllay book presentacion
byAlejoLevenstain1
 
PDF
Kate Scrivens-OECD-Community well-being and climate risks.pdf
byStatsCommunications
 
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
Presentation 1-PUM Singapore 2015- 2D.pptx
byWilliamChong71
 
Chapter 3 Reusable Technoldsadadsdsdasdogy.ppt
bytuananle9897
 
software define network authentication .pptx
byAhmedGouda93
 
BCS501 Module-05 Software Engineering and Project Management.pdf
byjaadu6918
 
[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...
byDataScienceConferenc1
 
Copy of First ENGR 100 Presentation.pptx
bygenshinds971
 
Natal SM 2025 - Latihan Liturgi Kelas 0 - 3 SD.pdf
byWenny63
 
[DSC Europe 25] Andy Cotgreave - Nothing is new in analytics.pptx
byDataScienceConferenc1
 
Time serise forcasting of non-durble consumer goods industry
byvkumaranand89
 
[DSC Europe 25] Vid Stimac - Policy Parsimony: Between Oversimplifying and Ov...
byDataScienceConferenc1
 
Safe_Spaces_Act_Modern_Presentation.pptx
byairesmijares2003
 
[DSC Europe 25] Dragana Ilic - AI for Big Data in Astronomy.pptx
byDataScienceConferenc1
 
[DSC Europe 25] Jim Sterne - Adopting Generative AI Capabilities Into the Ent...
byDataScienceConferenc1
 
Customer Segmentation: Seeing the Trees and the Forest Simultaneously - V2
bySK Palu
 
20251202 Frank_Melis Vittal_Prabhu Platform_Talk v3b.pdf
byhome
 
1. Prepare for the application of health survey.pptx
byDegaregeNibret
 
hindustan Unilever Limited Presentation - Sample
bymohitsawale3
 
Bicycle Project sostenible pllay book presentacion
byAlejoLevenstain1
 
Kate Scrivens-OECD-Community well-being and climate risks.pdf
byStatsCommunications
 

Introduction to Apache Tajo: Data Warehouse for Big Data

  • 1.
    Introduction to ApacheTajo: Data Warehouse for Big Data Jihoon Son / Gruter inc.
  • 2.
    About Me Jihoon Son(@jihoonson) Tajo project co-founder Committer and PMC member of Apache Tajo Research engineer at Gruter 2
  • 3.
    Outline About Tajo Features ofthe Recent Release Demo Roadmap 3
  • 4.
    What is Tajo? Tajo/ tάːzo / 타조 An ostrich in Korean The world's fastest two-legged animal 4
  • 5.
    What is Tajo? ApacheTop-level Project Big data warehouse system ANSI-SQL compliant Mature SQL features Various types of join, window functions Rapid query execution with own distributed DAG engine Low latency, and long running batch queries with a single system Fault-tolerance Beyond SQL-on-Hadoop Support various types of storage 5
  • 6.
    Tajo Master Catalog Server TajoMaster Catalog Server Architecture Overview 6 DBMS HCatalog Tajo Master Catalog Server Tajo Worker Query Master Query Executor Storage Service Tajo Worker Query Master Query Executor Storage Service Tajo Worker Query Master Query Executor Storage Service JDBC client TSQLWebUI REST API Storage Submit a query Manage metadataAllocate a query Send tasks & monitor Send tasks & monitor
  • 7.
    Who are UsingTajo? Use cases: replacement of commercial DW 1st telco in South Korea Replacement of long-running ETL workloads on several TB datasets Lots of daily reports about user behavior Ad-‐hoc analysis on TB datasets Benefits Simplified architecture for data analysis An unified system for DW ETL, OLAP, and Hadoop ETL Much less cost, more data analysis within same SLA Saved license fee of commercial DW 7
  • 8.
    Who are UsingTajo? Use cases: data discovery Music streaming service (26 million users) Analysis of purchase history for target marketing Benefits Interactive query on large datasets Data analysis with familiar BI tools 8
  • 9.
    Recent Release: 0.11 Featurehighlights Query federation JDBC-based storage support Self-describing data formats support Multi-query support More stable and efficient join execution Index support Python UDF/UDAF support 9
  • 10.
    Recent Release: 0.11 Today'stopic Query federation JDBC-based storage support Self-describing data formats support Multi-query support More stable and efficient join execution Index support Python UDF/UDAF support 10
  • 11.
  • 12.
    Your Data Your datamight be spread on multiple heterogeneous sites Cloud, DBMS, Hadoop, NoSQL, … 12 DBMS Application Cloud storage On-premise storage NoSQL
  • 13.
    Your Data Even ina single site, your data might be stored in different data formats 13 JSONCSV Parque t ORC Log ...
  • 14.
    Your Data How toanalyze distributed data? Traditionally ... 14 DBMSApplication Cloud storage On-premise storage NoSQL Global view ETL transform ● Long delivery ● Complex data flow ● Human-intensive
  • 15.
    Your Data withTajo Query federation 15 DBMSApplication Cloud storage On-premise storage NoSQL Global view ● Fast delivery ● Easy maintenance ● Simple data flow
  • 16.
    Storage and DataFormat Support 16 Data formats Storage types
  • 17.
    Create Table > CREATEEXTERNAL TABLE archive1 (id BIGINT, ...) USING text WITH ('text.delimiter'='|') LOCATION 'hdfs://localhost:8020/archive1'; > CREATE EXTERNAL TABLE user (user_id BIGINT, ...) USING orc WITH ('orc.compression.kind'='snappy') LOCATION 's3://user'; > CREATE EXTERNAL TABLE table1 (key TEXT, ...) USING hbase LOCATION 'hbase:zk://localhost:2181/uptodate'; > ... 17 Data formatStorage URI
  • 18.
    Create Table > CREATEEXTERNAL TABLE archive1 (id BIGINT, ...) USING text WITH ('text.delimiter'='|','text.null'='N','compression.codec'='org.apache.hadoop.io.compr ess.SnappyCodec','timezone'='UTC+9','text.skip.headerlines'='2') LOCATION 'hdfs://localhost:8020/tajo/warehouse/archive1'; > CREATE EXTERNAL TABLE archive2 (id BIGINT, ...) USING text WITH ('text.delimiter'='|','text.null'='N','compression.codec'='org.apache.hadoop.io.compr ess.SnappyCodec','timezone'='UTC+9','text.skip.headerlines'='2') LOCATION 'hdfs://localhost:8020/tajo/warehouse/archive2'; > CREATE EXTERNAL TABLE archive3 (id BIGINT, ...) USING text WITH ('text.delimiter'='|','text.null'='N','compression.codec'='org.apache.hadoop.io.compr ess.SnappyCodec','timezone'='UTC+9','text.skip.headerlines'='2') LOCATION 'hdfs://localhost:8020/tajo/warehouse/archive3'; > ... 18 Too tedious!
  • 19.
    Introduction to Tablespace Tablespace Registeredstorage space A tablespace is identified by an unique URI Configurations and policies are shared by all tables in a tablespace Storage type Default data format and supported data formats It allows users to reuse registered storage configurations and policies 19
  • 20.
    Tablespaces, Databases, andTables 20 Namespace Storage1 Storage2 ... ... ... Tablespace1 Tablespace2 Tablespace3 Physical space Table1 Table2 Table3 Database1 Database1 ...
  • 21.
    Tablespace Configuration { "spaces" :{ "warehouse" : { "uri" : "hdfs://localhost:8020/tajo/warehouse", "configs" : [ {'text.delimiter'='|'}, {'text.null'='N'}, {'compression.codec'='org.apache.hadoop.io.compress.SnappyCodec'}, {'timezone'='UTC+9'}, {'text.skip.headerlines'='2'} ] }, "hbase1" : { "uri" : "hbase:zk://localhost:2181/table1" } } } 21 Tablespace name Tablespace URI
  • 22.
    Create Table > CREATETABLE archive1 (id BIGINT, ...) TABLESPACE warehouse; 22 Tablespace nameData format is omitted. Default data format is TEXT. "warehouse" : { "uri" : "hdfs://localhost:8020/tajo/warehouse", "configs" : [ {'text.delimiter'='|'}, {'text.null'='N'}, {'compression.codec'='org.apache.hadoop.io.compress.SnappyCodec'}, {'timezone'='UTC+9'}, {'text.skip.headerlines'='2'} ] },
  • 23.
    Create Table > CREATETABLE archive1 (id BIGINT, ...) TABLESPACE warehouse; > CREATE TABLE archive2 (id BIGINT, ...) TABLESPACE warehouse; > CREATE TABLE archive3 (id BIGINT, ...) TABLESPACE warehouse; > CREATE TABLE user (user_id BIGINT, ...) TABLESPACE aws USING orc WITH ('orc.compression.kind'='snappy'); > CREATE TABLE table1 (key TEXT, ...) TABLESPACE hbase1; > ... 23
  • 24.
    HDFS HBase Tajo Worker Query Engine Storage ServiceHDFS handler TajoWorker Query Engine Storage ServiceHDFS handler Tajo Worker Query Engine Storage ServiceHBase handler Querying on Different Data Silos How does a worker access different data sources? Storage service Return a proper handler for underlying storage > SELECT ... FROM hdfs_table, hbase_table, ... 24
  • 25.
  • 26.
    jdbc_db1 tajo_db1 JDBC-based Storage Storageproviding the JDBC interface PostgreSQL, MySQL, MariaDB, ... Databases of JDBC-based storage are mapped to Tajo databases 26 Table 1 Table 2 Table 3 Table 1 Table 2 Table 3 tajo_db2 Table 1 Table 2 Table 3 … jdbc_db2 Table 1 Table 2 Table 3 … JDBC-based storage Tajo
  • 27.
    Tablespace Configuration { "spaces": { "pgsql_db1":{ "uri": "jdbc:postgresql://hostname:port/db1" "configs": { "mapped_database": "tajo_db1" "connection_properties": { "user": "tajo", "password": "xxxx" } } } } } 27 PostgreSQL database name Tajo database name Tablespace name
  • 28.
    Return to QueryFederation How to correlate data on JDBC-based storage and others? Need to have a global view of metadata across different storage types Tajo also has its own metadata for its data Each JDBC-based storage has own metadata for its data Each NoSQL storage has metadata for its data … 28
  • 29.
    Metadata Federation Federating metadataof underlying storage 29 DBMS metadata provider NoSQL metadata provider Linked Metadata Manager DBMS HCatalog Tajo catalog metadata provider Catalog Interface ● Tablespace ● Database ● Tables ● Schema names ...
  • 30.
    Querying on JDBC-basedStorage A plan is converted into a SQL string Query generation Diverse SQL syntax of different types of storage Different SQL builder for each storage type 30 Tajo Master Tajo Worker JDBC-based storage SELECT ... Query plan SELECT ...
  • 31.
    Operation Push Down Tajocan exploit the processing capability of underlying storage DBMSs, MongoDB, HBase, … Operations are pushed down into underlying storage Leveraging the advanced features provided by underlying storage Ex) DBMSs' query optimization, index, ... 31
  • 32.
    Example 1 32 SELECT count(*) FROM account ac,archive ar WHERE ac.key = ar.id and ac.name = 'tajo' account DBMS archive HDFS scan archive scan account ac.name = 'tajo' join ac.key = ar.id group by count(*) group by count(*) Full scan Result only Push operation
  • 33.
    Example 2 33 SELECT ac.name, count(*) FROM accountac GROUP BY ac.name account DBMS scan account group by count(*) Result only Push operation
  • 34.
  • 35.
    Self-describing Data Formats Somedata formats include schema information as well as data JSON, ORC, Parquet, … Tajo 0.11 natively supports self-describing data formats Since they already have schema information, Tajo doesn't need to store it aside Instead, Tajo can infer the schema at query execution time 35
  • 36.
    Create Table withNested Data Format { "title" : "Hand of the King", "name" : { "first_name": "Eddard", "last_name": "Stark"}} { "title" : "Assassin", "name" : { "first_name": "Arya", "last_name": "Stark"}} { "title" : "Dancing Master", "name" : { "first_name": "Syrio", "last_name": "Forel"}} > CREATE EXTERNAL TABLE schemaful_table ( title TEXT, name RECORD ( first_name TEXT, last_name TEXT ) ) USING json LOCATION 'hdfs:///json_table'; 36 Nested type
  • 37.
    How about ThisData? {"id":"2937257761","type":"ForkEvent","actor":{"id":1088854,"login":"CAOakleyII","gravatar_id":"","url":"https://api.github.com/users/CAOakleyII","avatar_url":"https://avatars.githubusercontent.com/ u/1088854?"},"repo":{"id":11909954,"name":"skycocker/chromebrew","url":"https://api.github.com/repos/skycocker/chromebrew"},"payload":{"forkee":{"id":38339291,"name":"chromebrew","full_na me":"CAOakleyII/chromebrew","owner":{"login":"CAOakleyII","id":1088854,"avatar_url":"https://avatars.githubusercontent.com/u/1088854?v=3","gravatar_id":"","url":"https://api.github.com/users/C AOakleyII","html_url":"https://github.com/CAOakleyII","followers_url":"https://api.github.com/users/CAOakleyII/followers","following_url":"https://api.github.com/users/CAOakleyII/following{/other_ user}","gists_url":"https://api.github.com/users/CAOakleyII/gists{/gist_id}","starred_url":"https://api.github.com/users/CAOakleyII/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/ users/CAOakleyII/subscriptions","organizations_url":"https://api.github.com/users/CAOakleyII/orgs","repos_url":"https://api.github.com/users/CAOakleyII/repos","events_url":"https://api.github.com/ users/CAOakleyII/events{/privacy}","received_events_url":"https://api.github.com/users/CAOakleyII/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/C AOakleyII/chromebrew","description":"Package manager for Chrome OS","fork":true,"url":"https://api.github.com/repos/CAOakleyII/chromebrew","forks_url":"https://api.github.com/repos/CAOakleyII/chromebrew/forks","keys_url":"https://api.github.com/repos/CAOa kleyII/chromebrew/keys{/key_id}","collaborators_url":"https://api.github.com/repos/CAOakleyII/chromebrew/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/CAOakleyII/chro mebrew/teams","hooks_url":"https://api.github.com/repos/CAOakleyII/chromebrew/hooks","issue_events_url":"https://api.github.com/repos/CAOakleyII/chromebrew/issues/events{/number}","event s_url":"https://api.github.com/repos/CAOakleyII/chromebrew/events","assignees_url":"https://api.github.com/repos/CAOakleyII/chromebrew/assignees{/user}","branches_url":"https://api.github.com /repos/CAOakleyII/chromebrew/branches{/branch}","tags_url":"https://api.github.com/repos/CAOakleyII/chromebrew/tags","blobs_url":"https://api.github.com/repos/CAOakleyII/chromebrew/git/blo bs{/sha}","git_tags_url":"https://api.github.com/repos/CAOakleyII/chromebrew/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/CAOakleyII/chromebrew/git/refs{/sha}","trees_url":"https:/ /api.github.com/repos/CAOakleyII/chromebrew/git/trees{/sha}","statuses_url":"https://api.github.com/repos/CAOakleyII/chromebrew/statuses/{sha}","languages_url":"https://api.github.com/repos/C AOakleyII/chromebrew/languages","stargazers_url":"https://api.github.com/repos/CAOakleyII/chromebrew/stargazers","contributors_url":"https://api.github.com/repos/CAOakleyII/chromebrew/cont ributors","subscribers_url":"https://api.github.com/repos/CAOakleyII/chromebrew/subscribers","subscription_url":"https://api.github.com/repos/CAOakleyII/chromebrew/subscription","commits_url": "https://api.github.com/repos/CAOakleyII/chromebrew/commits{/sha}","git_commits_url":"https://api.github.com/repos/CAOakleyII/chromebrew/git/commits{/sha}","comments_url":"https://api.gith ub.com/repos/CAOakleyII/chromebrew/comments{/number}","issue_comment_url":"https://api.github.com/repos/CAOakleyII/chromebrew/issues/comments{/number}","contents_url":"https://api.git hub.com/repos/CAOakleyII/chromebrew/contents/{+path}","compare_url":"https://api.github.com/repos/CAOakleyII/chromebrew/compare/{base}...{head}","merges_url":"https://api.github.com/repo s/CAOakleyII/chromebrew/merges","archive_url":"https://api.github.com/repos/CAOakleyII/chromebrew/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/CAOakleyII/chromebr ew/downloads","issues_url":"https://api.github.com/repos/CAOakleyII/chromebrew/issues{/number}","pulls_url":"https://api.github.com/repos/CAOakleyII/chromebrew/pulls{/number}","milestones_ url":"https://api.github.com/repos/CAOakleyII/chromebrew/milestones{/number}","notifications_url":"https://api.github.com/repos/CAOakleyII/chromebrew/notifications{?since,all,participating}","lab els_url":"https://api.github.com/repos/CAOakleyII/chromebrew/labels{/name}","releases_url":"https://api.github.com/repos/CAOakleyII/chromebrew/releases{/id}","created_at":"2015-07- 01T00:00:00Z","updated_at":"2015-06-28T10:11:09Z","pushed_at":"2015-06- 09T07:46:57Z","git_url":"git://github.com/CAOakleyII/chromebrew.git","ssh_url":"git@github.com:CAOakleyII/chromebrew.git","clone_url":"https://github.com/CAOakleyII/chromebrew.git","svn_url":" https://github.com/CAOakleyII/chromebrew","homepage":"http://skycocker.github.io/chromebrew/","size":846,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":false,"has_downl oads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master","public":true}},"public":true,"cre ated_at":"2015-07-01T00:00:01Z"} ... 37
  • 38.
    Create Schemaless Table >CREATE EXTERNAL TABLE schemaless_table (*) USING json LOCATION 'hdfs:///json_table'; That's all! 38 Allow any schema
  • 39.
    Schema-free Query Execution >CREATE EXTERNAL TABLE schemaful_table (id BIGINT, name TEXT, ...) USING text LOCATION 'hdfs:///csv_table; > CREATE EXTERNAL TABLE schemaless_table (*) USING json LOCATION 'hdfs:///json_table'; > SELECT name.first_name, name.last_name from schemaless_table; > SELECT title, count(*) FROM schemaful_table, schemaless_table WHERE name = name.last_name GROUP BY title; 39
  • 40.
    Schema Inference Table schemais inferred at query time Example 40 SELECT a, b.b1, b.b2.c1 FROM t; ( a text, b record ( b1 text, b2 record ( c1 text ) ) ) Inferred schemaQuery
  • 41.
  • 42.
  • 43.
  • 44.
    Roadmap 0.12 Improved Yarn integration Authenticationsupport JavaScript stored procedure support Scalar subquery support Hive UDF support 44
  • 45.
    Roadmap Next generation (beyond0.12) Exploiting modern hardware Approximate query processing Genetic query optimization And more … 45
  • 46.