Downloaded 19 times
![alluxio table attachdb <udb type> <udb uri> <udb db name>
associate an Alluxio database with an UDB database
alluxio table detachdb <db name>
remove the association from “attach”
alluxio table ls [<db name> [<table name>]]
display information in the catalog
alluxio table sync <db name>
synchronize the Alluxio catalog with the UDB metadata
How to Use Alluxio Catalog CLI
15](https://image.slidesharecdn.com/11genestructureddata-dataorchestrationsummitnov2019-191108233421/75/Alluxio-Innovations-for-Structured-Data-15-2048.jpg)
Uploaded byAlluxio, Inc.
Alluxio Innovations for Structured Data
Gene Pang from Alluxio presented on their new structured data management capabilities in Alluxio. Alluxio 2.1.0 includes preview components to integrate SQL engines like Presto with Alluxio's unified metadata catalog and caching. A demo showed Presto queries against a TPCDS dataset on S3 running over 3x faster when using Alluxio's transformations to coalesce and optimize the data format from CSV to Parquet and leverage Alluxio's caching. Future work may include additional connectors, formats, DDL/DML support and client APIs. Feedback from the user community is important to help guide the project.
More Related Content
Similar to Alluxio Innovations for Structured Data
Alluxio Innovations for Structured Data
- 1. DATA ORCHESTRATION SUMMIT 2019 AlluxioInnovations for Structured Data Gene Pang | Alluxio
- 2. Motivation Alluxio Structured DataManagement Developer Preview in Alluxio 2.1.0 Demo Conclusion Outline 2
- 3.
- 4. Common Alluxio UseCases 4 … … Unified Interface Unified Namespace Caching and Locality SQL Engines are popular
- 5. 5 Storage Systems SQLFrameworks Files/Objects Directories Raw Bytes Cost-efficiency Durability Tables Schemas Rows/Columns Compute-optimized Computation Impedance Mismatch Further Expand Benefits!
- 6. Benefits of AlluxioData Orchestration 6 Storage Systems SQL Frameworks Caching Unified Interface/Namespace Schema-Aware Optimizations Compute-Optimized Formats Physical Data Independence
- 7. 7 Demo 20 seconds 3seconds
- 8. DATA ORCHESTRATION SUMMIT 2019 Alluxio Structured DataManagement Integrating SQL Engines with Alluxio
- 9. Provide Structured DataAPIs Focus on how frameworks interact with data High-Level Philosophy 9 Cache Logical Data Access Focus on caching what frameworks want
- 10. Alluxio Structured DataManagement Alluxio Structured Data ManagementVision 10 Storage System Transformation Service Structured Data and Metadata Logical Data Access Layer Structured Data Client SQL Engine Engine
- 11. DATA ORCHESTRATION SUMMIT 2019 Developer Previewin Alluxio 2.1.0 Try out initial components!
- 12.
- 13. Alluxio Structured DataManagement Preview 13 Presto Alluxio Caching Service Alluxio Catalog Service AlluxioTransformation Service Hive Connector Alluxio Connector Hive Metastore Storage
- 14. Alluxio Catalog Service 14 AlluxioCatalog Service Hive Metastore Hive Under Database Functionality Manages metadata for structured data Abstracts other database catalogs as Under Database (UDB) Benefits Schema-aware optimizations Simple deployment
- 15. alluxio table attachdb<udb type> <udb uri> <udb db name> associate an Alluxio database with an UDB database alluxio table detachdb <db name> remove the association from “attach” alluxio table ls [<db name> [<table name>]] display information in the catalog alluxio table sync <db name> synchronize the Alluxio catalog with the UDB metadata How to Use Alluxio Catalog CLI 15
- 16. Tighter integration withPresto New plugin based on the Presto Hive connector Available in Alluxio 2.1.0 distribution Future: Merge connector into Presto codebase Alluxio Presto Connector 16
- 17.
- 18. Transformation Service 18 Transform datato be compute-optimized independent of the storage format Coalesce Format Conversion parquetcsv
- 19. alluxio table transform<db name> <table name> initiate a transformation on a table alluxio table transformStatus <transform id> display the status for a transformation How to UseTransformation CLI 19
- 20. Familiar Alluxio Concepts 20 StructuredData Filesystem Under Database Under Filesystem Attach/Detach database Mount/Unmount filesystem Transparent access of tables Transparent access of files Transparent caching portions of tables Transparent caching portions of files
- 21.
- 22. 2 isolated AWS10-node clusters Presto + Hive Metastore + S3 Data Presto + Alluxio + Hive Metastore + S3 Data TPCDS dataset on S3 CSV format ~10,000 files Demo 22
- 23. Attached existing Hivedatabase into Alluxio Catalog Alluxio Catalog served table metadata for Presto Transformed store_sales by coalescing and converting CSV to Parquet Demo Summary 23 Presto Without Alluxio 20s Alluxio Transformations 7s AlluxioTransformations With Caching 3s
- 24.
- 25. User community feedback/collaborationis important! Potential projects New UDB implementations (AWS Glue) More conversion formats (json) DDL/DML workloads (CREATETABLE, INSERT, etc.) New Client APIs for structured data (Arrow) Future Work 25
- 26. Try it out! Documentation Providefeedback Feature requests and issues in Github Alluxio/alluxio Developer Preview Available in Alluxio 2.1.0 26 ThankYou!
- 27.