Download as PDF, PPTX
Uploaded byAlluxio, Inc.
Data Orchestration for AI, Big Data, and Cloud
This document discusses the need for data orchestration across fragmented data environments. As more data is generated and stored across different storage systems and clouds, data silos have become inevitable. A data orchestration solution like Alluxio can abstract and orchestrate data across these silos, making data locally accessible to compute frameworks regardless of where the data is stored. Alluxio provides a unified view of data locations, enables data access from any application, and allows data to be burst elastically across clouds for compute. Many large companies are adopting data orchestration to improve data access, reduce costs, and gain more flexibility in their cloud environments.
In this document
Powered by AI
Introduction to data orchestration for AI, big data, and cloud by Haoyuan Li from Alluxio.
The increasing need for data access, generation, and evolving technologies lead to a fragmented data landscape.
Recognition that data silos are an inevitable challenge in managing modern data environments.
Discusses the limitations of managing data in a single data lake and the need for orchestration.
Illustrates the complexity of data silos across various data centers, clouds, and storage systems.
Highlights the need to abstract and orchestrate data across silos to enhance data workflows.
Benefits of orchestrating data in the cloud, including local data access and cost reduction.
Emphasizes the importance of data orchestration for AI, big data, and cloud environments.
Overview of various interfaces and drivers for data orchestration, enhancing compatibility.
Examines how data orchestration contributes to overall agility in data management.
Explains data orchestration strategies for compute bursting, particularly for financial institutions.
Details on open-source implementation of data orchestration and community involvement.
Highlights the trend of companies, including major internet firms, adopting data orchestration.
Encouragement to join the Alluxio open-source community and contribute to data orchestration.
More Related Content
Similar to Data Orchestration for AI, Big Data, and Cloud
Data Orchestration for AI, Big Data, and Cloud
- 1. Data Orchestration forAI, Big Data, and Cloud Haoyuan (HY) Li | Founder, Chairman, CTO | Alluxio haoyuan@alluxio.com | @haoyuan 2019-06-21 @ O’Reilly AI Beijing
- 2. The journey toa fragmented data world More people & teams need access to this data More data generated every day New compute & storage technologies created every 3-8 years
- 3.
- 4.
- 5. Data silos crossdata centers, regions, clouds HDFS HIVE HDFS Spark NFS TENSOR FLOW DATA IN DISPARATE STORAGE SYSTEMS OBJECT STORE PRESTO COMPUTE SPREAD ACROSS MANY DIFFERENT FRAMEWORKS WAN HDFS WAN S3 Spark AZURE PRESTO
- 6. Abstract & orchestratedata across data silos HDFS HIVE Spark NFS TENSOR FLOW DATA IN DISPARATE STORAGE SYSTEMS PRESTO COMPUTE SPREAD ACROSS MANY DIFFERENT FRAMEWORKS S3 SPARK DATA ORCHESTRATION DATA ORCHESTRATION DATA ORCHESTRATION DATA ORCHESTRATION DATA ORCHESTRATION ANY DATA APP DATA ORCHESTRATION
- 7. Data Orchestration forthe cloud Data Locality,Accessibility & Elasticity for AI & Big Data § Accelerate speed to insights with hot data made local to compute faster § Burst data elastically with compute anytime in any cloud environment § Reduce costs by time-consuming ETL and eliminating multiple persisted copies
- 8. Data Orchestration forthe AI, Big Data, and Cloud
- 9. Java File APIHDFS Interface S3 Interface REST APIFUSE Interface HDFS Driver Swift Driver S3 Driver NFS Driver Data Orchestration for the AI, Big Data, and Cloud
- 10. Data Orchestration forAgility
- 11. Data Orchestration forCompute Bursting Leading Hedge Fund
- 12. An Open SourceImplementation of Data Orchestration Started From UC Berkeley AMPLab 1000+ contributors & growing 4000+ Git Stars Apache 2.0 Licensed GitHub’s Top 100 Most Valuable Repositories Out of 96 Million Join the conversation on Slack slackin.alluxio.io
- 13. Companies Moving TowardsData Orchestration (Including 8 of the Top 10 Internet Companies in China) Read More
- 14. Embracing Data Silos– the Data Orchestration Approach Welcome to join the Alluxio Open Source Community! www.alluxio.io | @alluxio | slackin.alluxio.io