[OpenStack Day in Korea 2015] Track 2-6 - Apache Tajo on Swift
•
2 likes•849 views
Jihoon Son presents Apache Tajo, an open source data warehouse system that supports SQL queries. Tajo can run on OpenStack Swift object storage without any code modifications, addressing the need for SQL queries on Swift data. Son demonstrates how Tajo integrates with Swift, discusses configurations, and addresses the data locality problem. Advanced integration techniques like a location-aware computing model are presented to improve performance when Tajo and Swift clusters are co-located. The roadmap includes specialized storage layers for Swift and support for additional storage like Cinder and Ceph block storage.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to [OpenStack Day in Korea 2015] Track 2-6 - Apache Tajo on Swift
Similar to [OpenStack Day in Korea 2015] Track 2-6 - Apache Tajo on Swift (20)
More from OpenStack Korea Community
More from OpenStack Korea Community (20)
Recently uploaded
Recently uploaded (20)
[OpenStack Day in Korea 2015] Track 2-6 - Apache Tajo on Swift
- 1. Apache Tajo on Swift Bringing SQL to the OpenStack World Jihoon Son Apache Tajo PMC member
- 2. Who am I ● Jihoon Son ○ Ph.D candidate (Computer Science & Engineering, 2010.3 ~) ○ Apache Tajo PMC and Committer (2014.5.1 ~) ○ Mentor of Google Summer of Code (2013) ● Contacts ○ Email: jihoonson AT apache.org ○ LinkedIn: https://www.linkedin.com/in/jihoonson
- 3. Outline ● OpenStack Swift ● Apache Tajo ● Tajo on Swift ● Demo ● Our Roadmap
- 4. OpenStack Swift ● Popular object storage ○ Images, videos, logs, ... ● Enterprises store objects on Swift to provide their services ○ Usually private clusters
- 5. SQL on Swift ● Data analysis is important to improve the quality of their services ○ SQL is one of the most powerful and popular query language ● Many enterprise data analysis tools relying on SQL ○ OLAP, visualization, data mining, … ● Need for using SQL on Swift
- 6. Apache Tajo ● Scalable, efficient, and fault-tolerant data warehouse system ○ Support SQL standards compliance ○ Efficient batch execution and interactive ad-hoc analysis ■ Low latency and high throughput ■ No use of MapReduce ○ No single point of failure
- 7. Apache Tajo ● Active open source project ○ 18 committers and 16 contributors ○ Activity summary
- 8. Apache Tajo Pluggable Storage Layer ... MasterMasterTajo Master Tajo Worker Tajo Worker Tajo Worker Tajo Worker ...
- 9. Tajo on Swift Pluggable Storage Layer MasterMasterTajo Master Tajo Worker Tajo Worker Tajo Worker Tajo Worker ... ... Swift
- 10. Tajo on Swift ● No need to modify code of Tajo and Swift ○ Tajo can access Swift with the Hadoop-openstack library ■ But, doesn’t need to install or run Hadoop ○ Just use it Swift Network
- 11. Tajo on Swift ● Configuration highlights ○ Swift configuration ■ Need the keystone authentication for the Hadoop ■ No additional configurations ○ HDFS configuration ■ Different cloud providers support ● Key name pattern fs.swift.service.${provider}
- 12. Tajo on Swift ● Configuration highlights ○ Swift configuration ■ Need the keystone authentication for the HDFS client ■ No additional configurations ○ HDFS configuration ■ Different cloud providers support ● Key name pattern fs.swift.service.${provider}
- 13. Tajo on Swift ● Data locality problem Worker Storage Node Interconnection Network Node A Worker Node B Storage Node Significant Network Overhead
- 14. Tajo on Swift ● Data locality problem Worker Storage Node Interconnection Network Node A Worker Node B Storage Node
- 15. Advanced Integration ● List endpoints middleware ○ Providing the location information of objects, accounts or containers ■ Tajo workers can directly access each object ○ Example
- 16. Advanced Integration ● List endpoints middleware ○ Swift configuration ○ ○ ○ Hadoop configuration
- 17. Advanced Integration ● Location-aware computing ○ Moving the processing close to the data ■ Avoiding the performance degradation due to the data transfer over the network ○ Important issue when Tajo and Swift share the same cluster
- 19. Storage Node Location-aware Computing 1. Getting object locations from the ring Query Master MasterMasterProxy Server Get object locations Storage Node Storage Node
- 20. Location-aware Computing 2. Assigning tasks based on object locations Query Master Worker Worker Worker ... Storage Node Storage Node Storage Node ... Assign tasks close to the object Directly read object data
- 21. Demo
- 22. Our Roadmap ● Storage layer specialized for Swift ● Block storage support ○ Cinder and Ceph ● Provisioning Tajo clusters ○ Sahara ○ Heat, TOSCA