Efficient In-situ Processing of Various Storage Types on Apache Tajo
•
2 likes•390 views
This document discusses Apache Tajo, an open source data warehouse system that supports efficient in-situ processing of various storage types on Hadoop. It provides an overview of Tajo, describes its support for different storage formats like HDFS, S3, HBase and JDBC. The motivation, design considerations and current status of supporting various storages are explained. Tajo achieves this through a pluggable storage layer, query rewrite rules, and pushing down operations to underlying storages for optimization. The document encourages involvement in the project.
More Related Content
What's hot
What's hot (20)
Similar to Efficient In-situ Processing of Various Storage Types on Apache Tajo
Similar to Efficient In-situ Processing of Various Storage Types on Apache Tajo (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Efficient In-situ Processing of Various Storage Types on Apache Tajo
- 1. Efficient In-situ Processing of Various Storage Types on Apache Tajo Hadoop Summit 2015 San Jose Hyunsik Choi, Gruter Inc.
- 2. Agenda • Tajo Overview • Various Storage Support • Motivation • Design Consideration • What we did/are doing
- 3. An overview of Apache Tajo
- 4. Tajo: A Data Warehouse System • Data Warehouse System • Apache Top-level project • Low latency, and long running batch queries in a single system • ~100 ms up to several hours • Fault tolerance • Features • ANSI SQL compliance • Mature SQL features: Joins, Group by, Sort, Multiple distinct aggregations and Window function • Partitioned table support • Java/Python UDF support • JDBC driver and Java-based asynchronous API • SQL data type and Nested type support • Direct JSON support
- 5. Master Server TajoMaster Slave Server TajoWorker QueryMaster Local Query Engine StorageManager Local FileSystem HDFS Client JDBC SQL Shell Web UI Slave Server TajoWorker QueryMaster Local Query Engine StorageManager Local FileSystem HDFS Slave Server TajoWorker QueryMaster Local Query Engine StorageManager Local FileSystem HDFS CatalogStore DBMS (MySQL, ..) Hive Meta StoreSubmit a query Manage metadata Allocate a query send tasks & monitor send tasks & monitor Tajo Overall Architecture
- 6. Background: Query Optimization Phases
- 7. BLK 1 BLK 5 BLK 3 BLK 4 BLK 2 BLK 6 Task Assigning with Locality Worker Worker Worker HDFS Cluster Node1 Node2 Node3 Tajo Cluster… … … • Each task is assigned to a node according to its locality. Background: Task Execution
- 8. • Physical operators are assembled into a tree and their execution pipelined in the same machine. • Leaf operators must be scanners. • Tajo provides abstraction scanner, allowing to read different physical tables. Background: Local Execution
- 10. Motivation • Unified Interface • Data Integration • In-situ Processing HDFS NoSQL S3 Openstack Swift Apache Tajo
- 11. Sequence File RCFile Protocol Buffer Datasets stored in Various Formats/Storages
- 12. Design Considerations • More Storage Properties • Splittable, compressible (codecs), indexable, seekable, projectable, aggregatable, … • Query Optimization • Pluggable Storage and Data Format • More operation pushdown
- 13. Sequence File RCFile Protocol Buffer Separation between Storage and Format Data Formats Storage Types
- 14. Relationships between Storage and Format Storages Data Format Text RCFile Parquet Avro Hbase Serialization Protobuf Local File System Swift ….. JSON . . . .
- 15. Tablespace • Tablespace • Each table space is identified by a URI. • Hdfs://host:port/warehouse, hbase:zk://quorum1:2171, quorum2:2171, … • All tables in the same tablespace shares the same configuration. • URI scheme indicates storage type. • Hdfs, hbase, jdbc, … • Multiple tablespaces is possible in single storage namespace. • HDFS-2832: Enable support for heterogeneous in HDFS. • e.g., • /warehouse/ (disk) • /today/ (ssd)
- 16. Storage Configuration Storage Type Name and URI scheme Storage Handler Class
- 18. Format Configuration Format names The relationship between formats and storages
- 19. CREATE Table using Tablespace CREATE TABLE uptodate (key TEXT, …) TABLESPACE hbase1; CREATE TABLE archive (l_orderkey bigint, …) TABLESPACE warehouse USING text WITH (‘text.delimiter’ = ‘|’); Tablespace Name Format name
- 20. Storage Layer Access over Query Lifecycle Query Planning Query Executor Running Task Completed Task Failed Task Query Master - guessTableVolume() - validateSchema() - getStorageProperty() - getFormatProperty() - … - createTable() - purgeTable() - prepareTable() - getScanner() - getAppender() ….. - getSplits() - commitTable() - rollbackTable() - getScanner() - getAppender()
- 21. Query Rewrite for Specific Storages CREATE TABLE hbase_table (key TEXT, …) INSERT INTO hbase_table SELECT id, name, … SCAN Table Write SCAN Table Write Sort Logical Plan Rewrite HFileHandler
- 22. Operation Push Down SELECT X, SUM(Y) FROM table1 WHERE x > 100 GROUP BY x Underlying Storage Filter and Projection can be pushed down into Underlying storages (like RDBMS, Hbase, Elasticsearch, …)
- 23. Current Status • Storages: • HDFS support • Amazon S3 and Openstack Swift • Hbase Scanner and Writer - Hfile and Put Mode • JDBC-based Scanner and Writer (Working) • Kafka Scanner (Patch Available) • Elastic Search (Patch Available) • Data Formats • Text, JSON, RCFile, SequenceFile, Avro, Parquet, and ORC (Patch Available)
- 24. Get Involved! • We are recruiting contributors! • General • http://tajo.apache.org • Getting Started • http://tajo.apache.org/docs/0.10.0/getting_started.html • Downloads • http://tajo.apache.org/downloads.html • Jira – Issue Tracker • https://issues.apache.org/jira/browse/TAJO • Join the mailing list • dev-subscribe@tajo.apache.org • issues-subscribe@tajo.apache.org
- 25. Q&A
Editor's Notes
- Hi, I’m a pmc member of Apache Tajo project and I’m also research director of Gruter Inc. Today, I’m going to present The various strorage support in Tajo project. As you know, Tajo is a SQL on hadoop system. Especially, its exact position is data warehouse system. One of the key feature is to integrate different data sources and warehouse them. For them, different storage type support is necessary. The early Tajo also supports various storage types. But while we expanding storage types, we faced some limitation. In this presentation, I’ll present how Tajo solve this challenge and how Tajo support different storage types.
- This is an agenda of Today talk. Firstly, I’ll give an overview of Tajo project. We’ll talk about its overall architecture, how query works, how query is planned. Next, we will talk about how Tajo support various storage support.
- One of the distinguished features is that you can execute a low latency and long-running batch queries in a single system. Internally, It’s smart optimizater makes good use of various query optimization opportunities for various workloads. And also, it has ANSI SQL compliance, Mature SQL feature, Java/Python UDF support, JDBC, and nested data type. JSON support. Tajo is already a production-ready system.
- This figure shows an overall architecture of Tajo. One Tajo cluster consists of one master and multiple Tajo workers. Tajo master is responsible for coordinating Tajo workers and planning. When a query is issued, TajoMAster determines if this query needs a distributed execution. Some queries like DDL can be executed instantly in TajoMaster. Otherwise, a query is forwarded to free TajoWorker. Then, the query master is launched in the worker and it makes a number tasks from the query. Then tasks are disseminated into a number of nodes and executed on them. Tajo also has its own catalog store. This catalog store supports pluggable driver for persistent storage. Tajo can use Hive meta store as a persistent storage. In this case, Tajo can see the same table to Hive. Users can directly process Hive tables with Tajo.
- This figure shows how query is optimized. Tajo uses multiple well-defined optimization phases. Each phase tries to find the best answer with given information.
- Similar to other Hadoop-based processing systems, each leaf task which scans a table fragment is assigned to nodes according to the task locality.
- For local execution, a physical operators is assembled into a tree. Leaf operator scans a table fragment.
- Why? What challenges? How we Tajo sees various storages as table.
- Why we should support various storages in analytical SQL system? I think that there are three reasons. I also believe many users also think so. The first reason is an unified interface. SQL is easy and very popular interface. In particular, SQL has already various tools like BI tools and visualization tools. If you can use SQL on storages, you don’t need to pay additional cost for learning and tools. The next reason is data integration. Recently, most companies use various storages. For updatable data and point queries, NoSQL or RDBMS is widely used. For archiving data, HDFS or S3 is used. Users want to see a global view of data sets. In this case, analytical processing requires data integration among data sources, that is, joining multiple data sources. BTW, Analytical SQL systems are the most proper systems for join. The third reason is in-situ processing. it is common for users to perform ETL workload, causing heavy reading and processing costs. By the way, many storage systems have some query processing capability like filter and projection. In particular, RDBMS system can do more complex processing. This capability is much more efficient than what external system can do. So, if we use those capabilities, data archiving or ETL would be more efficient.
- In Hadoop eco system and legacy systems, there are many de-factor standard data formats. Our goal is to support them seemlessly.
- It’s just not reading and writing problems. In order to fully exploit storages, we need to consider more than just reading and writing. In particular, Tajo is a SQL system. One of the key feature of SQL system is query optimization. For them, storage layer should expose more physical properties like splittable, compressible, indexable, projectable, and sorted keys. Then, query optimizer should exploit physical properties. Also, we need pluggable storage and data format system, allowing users to add them without any code modification. Also, we need to enable storage layer to accept operation pushdown for in-situ processing.
- Firstly, we explicitly separate the storage concept into storage and data format. This separation allows us to easily describe more properties of specific storages and data formats. It also makes modularization much easier. For example, the scan throughput of RCFile is not bad in HDFS or Local File system. But, in S3, RCFile is too slow due to expensive random access cost. Another example is Hfile of Hbase. Tajo directly HFile for hbase loading. Even though Hfile is stored in HDFS, it requires sorted data set. We wanted these kinds of properties can be described.
- Then, we improve Tajo storage layer to recognize the relationships between storages and data formats. Some data formats like Text, Parquet are shared among file-based storage types.
- We also adopt Tablespace concept. HDFS-2832 introduced heterogeneous storage in HDFS. It allows different paths in a single namespace to use different storage media. For example, with this feature, we can bind warehouse directory to disk and today directory to ssd disk. Tablespace allows Tajo to distinguish directories in the same namespace as different spaces.
- For new concept, we need new configuration system. This configuration defines storages. This is a storage type name and it is also used for URI scheme. This class will handle all storage instances with URIs using hdfs scheme.
- This configuration describes tablespaces. This is the tablespace name. This table uri starts with hdfs. So, HDFS storage handler as I mentioned before will handle this tablespace.
- This configuraiton describes file formats. This is a format name. It means that Avro format can be used in hdfs, file, and s3 storages.
- In order to allow users to easily use tablespace concept, we also added tablespace clause. With tablespace clause, users can easily specify a tablespace for each table.
- Then, we also investigated the query lifecycle and access points to storage layer. Planning requires storage support. In Tajo, query optimizer uses storage layer to guess input table volume or get table properties. Especially, the table volume is commonly used for join optimization when there is no statistic information. QueryExecutor in TajoMaster determines if each query is executed in instantly executed in TajoMaster or executed in a distributed manner. DDL queries like CREATE TABLE, and DROP TABLE also need storage layer support. They require storage to make or purge storage space for a table. When a query is executed across a number of cluster nodes, TajoMaster forwards a query to QueryMaster, which will be launched instantly in a worker. QueryMaster makes splits from input data sources. Making splits also need storage layer support because it can be different according to storage type and file formats. For example, Splitting Text file in HDFS and S3 can be different. Then, tasks are disseminated across a number of workers. Each task scans input data and writes output data on specific storage. In this point, they also requires storage support. Tajo supports fault tolerance, The task failure is handled. So, only completed tasks are committed into the final output directory. We derived storage access pattern from this lifecycle and clearly defined storage layer APIs.
- While we are investigating various storages, we also realized that more APIs are requires for them. An example is the rewrite rule injection. As I mentioned ealier, Tajo supports direct Hfile write, requiring sorted table write. Hbase requires a written data sorted in an ascending order of row keys. In this case, sort operation will be injected before table writing.
- We are also expecting that aggregation also can be pushed down into underlying storage like RDBMS. This work is still working.