Alluxio Online Office Hours January 14, 2020 We introduce the concepts and components of Alluxio Structured Data Management, and go through a demo with Presto.

1. Alluxio Office Hour:
Alluxio Structured Data Management
Gene Pang | Alluxio

2. Motivation
Alluxio Structured Data Management
Developer Preview in Alluxio 2.1
Demo
Conclusion
Outline
2

3. Motivation

4. Common Alluxio Use Cases
4
…
…
Unified Interface
Unified Namespace
Caching and Locality
SQL Engines are popular

5. 5
Storage Systems SQL Frameworks
Files/Objects
Directories
Raw Bytes
Storage
Optimized
Tables
Schemas
Rows/Columns
Compute
Optimized
Impedance Mismatch
Further Expand Benefits!

6. Benefits of Alluxio Data Orchestration
6
Storage
Systems
SQL
Frameworks
Caching
Unified Interface/Namespace
Schema-Aware Optimizations
Compute-Optimized Formats
Physical Data Independence

7. Alluxio
Structured Data Management
Integrating SQL Engines with Alluxio

8. Provide Structured Data APIs
Focus on how frameworks interact with data
High-Level Philosophy
8
Cache Logical Data Access
Focus on caching what frameworks want

9. Alluxio Structured Data Management
Alluxio Structured Data Management
9
Storage
System
Transformation
Service
Structured Data
and Metadata
Logical Data
Access Layer
Structured
Data Client
SQL
Engine
Engine

10. Developer Preview in Alluxio 2.1
Try out initial components!

11. Target Environment
11
Presto
Hive
Connector
Hive
Metastore
Storage

12. Alluxio Structured Data Management
12
Presto
Alluxio Caching
Service
Alluxio Catalog
Service
AlluxioTransformation
Service
Hive
Connector
Alluxio
Connector
Hive
Metastore
Storage

13. Alluxio Catalog Service
13
Alluxio Catalog Service
Hive Metastore
Hive Under Database
Functionality
Manages metadata for structured data
Abstracts other database catalogs as
Under Database (UDB)
Benefits
Schema-aware optimizations
Simple deployment

14. Tighter integration with Presto
New plugin based on the Presto Hive connector
Available in Alluxio 2.1 distribution
In Progress: Merging connector into Presto codebase
Alluxio Presto Connector
14

15. Transformation Service
15
Transform data to be compute-optimized
independent from storage-optimized format
Coalesce Format Conversion
parquetcsv

16. Demo!

17. 2 isolated AWS 10-node clusters
Presto + Hive Metastore + S3 Data
Presto + Alluxio + Hive Metastore + S3 Data
TPCDS sample dataset on S3
~10,000 CSV files
Demo
17

18. Attached existing Hive database into Alluxio Catalog
Alluxio Catalog served table metadata for Presto
Transformed store_sales by coalescing and converting CSV to Parquet
Demo Summary
18
Presto Without
Alluxio
20s
Alluxio
Transformations
7s
AlluxioTransformations
With Caching
3s

19. Conclusion

20. User community feedback/collaboration is important!
Future projects
New UDB implementations (AWS Glue)
More conversion formats (json)
DDL/DML workloads (CREATETABLE, INSERT, etc.)
New Client APIs for structured data (Arrow)
Future Work
20

21. Try it out!
Documentation
Provide feedback
Feature requests and issues in Github Alluxio/alluxio
Developer Preview Available in Alluxio 2.1
21
ThankYou!

22. 22