Jihoon Son presents Apache Tajo, an open source data warehouse system that supports SQL queries. Tajo can run on OpenStack Swift object storage without any code modifications, addressing the need for SQL queries on Swift data. Son demonstrates how Tajo integrates with Swift, discusses configurations, and addresses the data locality problem. Advanced integration techniques like a location-aware computing model are presented to improve performance when Tajo and Swift clusters are co-located. The roadmap includes specialized storage layers for Swift and support for additional storage like Cinder and Ceph block storage.

1. Apache Tajo on Swift
Bringing SQL to the OpenStack World
Jihoon Son
Apache Tajo PMC member

2. Who am I
● Jihoon Son
○ Ph.D candidate (Computer Science & Engineering,
2010.3 ~)
○ Apache Tajo PMC and Committer (2014.5.1 ~)
○ Mentor of Google Summer of Code (2013)
● Contacts
○ Email: jihoonson AT apache.org
○ LinkedIn: https://www.linkedin.com/in/jihoonson

3. Outline
● OpenStack Swift
● Apache Tajo
● Tajo on Swift
● Demo
● Our Roadmap

4. OpenStack Swift
● Popular object storage
○ Images, videos, logs, ...
● Enterprises store objects on Swift to provide
their services
○ Usually private clusters

5. SQL on Swift
● Data analysis is important to improve the quality
of their services
○ SQL is one of the most powerful and popular query
language
● Many enterprise data analysis tools relying on
SQL
○ OLAP, visualization, data mining, …
● Need for using SQL on Swift

6. Apache Tajo
● Scalable, efficient, and fault-tolerant data
warehouse system
○ Support SQL standards compliance
○ Efficient batch execution and interactive ad-hoc
analysis
■ Low latency and high throughput
■ No use of MapReduce
○ No single point of failure

7. Apache Tajo
● Active open source project
○ 18 committers and 16 contributors
○ Activity summary

8. Apache Tajo
Pluggable Storage Layer
...
MasterMasterTajo
Master
Tajo
Worker
Tajo
Worker
Tajo
Worker
Tajo
Worker
...

9. Tajo on Swift
Pluggable Storage Layer
MasterMasterTajo
Master
Tajo
Worker
Tajo
Worker
Tajo
Worker
Tajo
Worker
...
...
Swift

10. Tajo on Swift
● No need to modify code of Tajo and Swift
○ Tajo can access Swift with the Hadoop-openstack
library
■ But, doesn’t need to install or run Hadoop
○ Just use it
Swift
Network

11. Tajo on Swift
● Configuration highlights
○ Swift configuration
■ Need the keystone authentication for the Hadoop
■ No additional configurations
○ HDFS configuration
■ Different cloud providers support
● Key name pattern
fs.swift.service.${provider}

12. Tajo on Swift
● Configuration highlights
○ Swift configuration
■ Need the keystone authentication for the HDFS client
■ No additional configurations
○ HDFS configuration
■ Different cloud providers support
● Key name pattern
fs.swift.service.${provider}

13. Tajo on Swift
● Data locality problem
Worker
Storage
Node
Interconnection Network
Node A
Worker
Node B
Storage
Node
Significant
Network
Overhead

14. Tajo on Swift
● Data locality problem
Worker
Storage
Node
Interconnection Network
Node A
Worker
Node B
Storage
Node

15. Advanced Integration
● List endpoints middleware
○ Providing the location information of objects,
accounts or containers
■ Tajo workers can directly access each object
○ Example

16. Advanced Integration
● List endpoints middleware
○ Swift configuration
○
○
○ Hadoop configuration

17. Advanced Integration
● Location-aware computing
○ Moving the processing close to the data
■ Avoiding the performance degradation due to the data
transfer over the network
○ Important issue when Tajo and Swift share the same
cluster

18. Location-aware Computing
Storage
Node
Storage
Node
Storage
Node
Query
Master
MasterMasterProxy
Server
Tajo
Worker
Tajo
Worker
Tajo
Worker
Data
location
Data
Swift Cluster Tajo Cluster

19. Storage
Node
Location-aware Computing
1. Getting object locations from the ring
Query
Master
MasterMasterProxy
Server
Get object locations
Storage
Node
Storage
Node

20. Location-aware Computing
2. Assigning tasks based on object locations
Query
Master
Worker Worker Worker ...
Storage
Node
Storage
Node
Storage
Node
...
Assign tasks
close to the object
Directly read object data

21. Demo

22. Our Roadmap
● Storage layer specialized for Swift
● Block storage support
○ Cinder and Ceph
● Provisioning Tajo clusters
○ Sahara
○ Heat, TOSCA

23. Thanks!
http://tajo.apache.org/