Uploaded byGruter
Efficient In‐situ Processing of Various Storage Types on Apache Tajo
The document discusses Apache Tajo, an open source data warehouse system that supports efficient in-situ processing of various storage types. It describes Tajo's architecture, how it supports different storage backends like HDFS, S3, HBase, and SQL databases. The document outlines Tajo's design for separating storage from data format and using tablespaces to manage different storage configurations. It also covers query optimization, operation pushdown, and status of current storage and format support in Tajo.
More Related Content
Similar to Efficient In‐situ Processing of Various Storage Types on Apache Tajo
![[DSC Europe 25] Nikola Rajovic - Hardware Technologies Under the Hood: RISC-V...](https://cdn.slidesharecdn.com/ss_thumbnails/o2gptrmtoyqndgoshwgq-dsc2025-tenstorrent-rajovic-251205090438-814685f5-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Jim Sterne - Adopting Generative AI Capabilities Into the Ent...](https://cdn.slidesharecdn.com/ss_thumbnails/sxhpofuorcagxsaulkmt-3-251204082258-7e66bc48-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dusan Jovicic - AI Story: From on-prem to cloud and back agai...](https://cdn.slidesharecdn.com/ss_thumbnails/8kp49m6uq22ifnbwhfnk-2-251205085715-964d11a6-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Boris Perkovic - Lost in performance.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/uq5hrp7vsuahqkxzifux-1-251204082258-fd2ee09d-thumbnail.jpg?width=640&height=640&fit=bounds)
Efficient In‐situ Processing of Various Storage Types on Apache Tajo
- 1. Efficient In-‐situ Processing of Various Storage Types on Apache Tajo Hadoop Summit 2015 San Jose Hyunsik Choi, Gruter Inc.
- 2. Agenda • Tajo Overview •Various Storage Support • Motivation • Design Consideration • What we did/are doing
- 3. An overview of Apache Tajo
- 4. Tajo: A Data Warehouse System • Data Warehouse System • Apache Top-‐level project • Low latency, and long running batch queries in a single system • ~100 ms up to several hours • Fault tolerance • Features • ANSI SQL compliance • Mature SQL features: Joins, Group by, Sort, Multiple distinct aggregations and Window function • Partitioned table support • Java/Python UDF support • JDBC driver and Java-‐based asynchronous API • SQL data type and Nested type support • Direct JSON support
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34. BLK 1 BLK 5 BLK3 BLK 4 BLK 2 BLK 6 Task Assigning with Locality Worker Worker Worker HDFS Cluster Node1 Node2 Node3 Tajo Cluster… … … • Each task is assigned to a node according to its locality. Background: Task Execution
- 35. • Physical operators are assembled into a tree and their execution pipelined in the same machine. • Leaf operators must be scanners. • Tajo provides abstraction scanner, allowing to read different physical tables. Background: Local Execution
- 36.
- 37. Motivation • Unified Interface • Data Integration • In-‐situ Processing HDFS NoSQL S3 Openstack Swift Apache Tajo
- 38. Sequence File RCFile Protocol Buffer Datasets stored in Various Formats/Storages
- 39. Design Considerations • More Storage Properties • Splittable, compressible (codecs), indexable, seekable, projectable, aggregatable, … • Query Optimization • Pluggable Storage and Data Format • More operation pushdown
- 40. Sequence File RCFile Protocol Buffer Separation between Storage and Format Data Formats Storage Types
- 41. Relationships between Storage and Format Storages Data Format Text RCFile Parquet Avro Hbase Serialization Protobuf Local File System Swift ….. JSON . . . .
- 42. Tablespace • Tablespace • Each table space is identified by a URI. • Hdfs://host:port/warehouse, hbase:zk://quorum1:2171,quorum2:2171, … • All tables in the same tablespace shares the same physical configuration. • URI scheme indicates storage type. • Hdfs, hbase, jdbc, … • Multiple tablespaces is possible in single storage namespace. • HDFS-‐2832: Enable support for heterogeneous in HDFS. • e.g., • /warehouse/ (disk) • /today/ (ssd)
- 43. Storage Configuration Storage Type Name and URI scheme Storage Handler Class
- 44.
- 45. Format Configuration Format names The relationship between formats and storages
- 46. CREATE Table using Tablespace CREATE TABLE uptodate (key TEXT, …) TABLESPACE hbase1; CREATE TABLE archive (l_orderkey bigint, …) TABLESPACE warehouse USING text WITH (‘text.delimiter’ = ‘|’); Tablespace Name Format name
- 47. Storage Layer Access over Query Lifecycle Query Planning Query Executor Running Task Completed Task Failed Task Query Master -‐ guessTableVolume() -‐ validateSchema() -‐ getStorageProperty() -‐ getFormatProperty() -‐ … -‐ createTable() -‐ purgeTable() -‐ prepareTable() -‐ getScanner() -‐ getAppender() ….. -‐ getSplits() -‐ commitTable() -‐ rollbackTable() -‐ getScanner() -‐ getAppender()
- 48. Query Rewrite for Specific Storages CREATE TABLE hbase_table (key TEXT, …) INSERT INTO hbase_table SELECT id, name, … SCAN Table Write SCAN Table Write Sort Logical Plan Rewrite HFileHandler
- 49. Operation Push Down SELECT X, SUM(Y) FROM table1 WHERE x 100 GROUP BY x Underlying Storage Filter and Projection can be pushed down into Underlying storages (like RDBMS, Hbase, Elasticsearch, …)
- 50. Current Status • Storages: •HDFS support • Amazon S3 and Openstack Swift • Hbase Scanner and Writer -‐ Hfile and Put Mode • JDBC-‐based Scanner and Writer (Working) • Kafka Scanner (Patch Available) • Elastic Search (Patch Available) • Data Formats • Text, JSON, RCFile, SequenceFile, Avro, Parquet, and ORC (Patch Available)
- 51. Get Involved! • We are recruiting contributors! • General • http://tajo.apache.org • Getting Started • http://tajo.apache.org/docs/0.10.0/getting_started.html • Downloads • http://tajo.apache.org/downloads.html • Jira – Issue Tracker • https://issues.apache.org/jira/browse/TAJO • Join the mailing list • dev-‐subscribe@tajo.apache.org • issues-‐subscribe@tajo.apache.org
- 52.