Downloaded 23 times
Uploaded byAlluxio, Inc.
Achieving compute and storage independence for data-driven workloads
Alluxio provides a unified interface to access data across multiple storage systems, allowing compute and storage to scale independently for data-driven applications. It uses a virtual unified file system with a global namespace and server-side API translation to abstract data location and access. Alluxio intelligently manages data placement across memory, SSDs and HDDs using multi-tier caching for local performance on remote data. This allows flexible deployment of compute like Spark on any cloud while keeping data fully controlled on-premises. Alluxio is seeing wide adoption with many large production deployments handling thousands of nodes. Upcoming features include POSIX API support and preview of version 2.0.
More Related Content
Similar to Achieving compute and storage independence for data-driven workloads
Achieving compute and storage independence for data-driven workloads
- 1. Achieving compute andstorage independence for data-driven workloads Two SigmaOpen Source Meetup Wenbo Zhao | Software Engineer |Two Sigma Bin Fan | Founding Engineer &VP, Open Source | Alluxio
- 2. Agenda Why storage-independent compute? AlluxioOverview Demo: Spark + Alluxio + S3 Future work
- 3. A Stack toEnable Cost-Effective Elastic Compute Spark/Presto/TF.. Alluxio HDFS Public Cloud On-Prem Burst compute workload to Cloud Data in a fully controlled environment Improved Data Locality
- 4. The Alluxio Story Projectstarted asTachyon, at the UC Berkley’s AMP Lab by then Ph.D. student & now Alluxio CEO, Haoyuan (H.Y.) Li. 2014 2015 Open Source project established & company to commercialize Alluxio founded Goal: Unify Data at Memory Speed for data driven applications such as Big Data Analytics, ML and AI. 2018 Top10 Hottest Data Storage Startup
- 5. From mainframes toBig Data Moving from tightly integrated to loosely integrated architectures Application, processing, data storage and hardware - All-in-one tightly coupled Client server architecture drives application separation. Processing and data storage still tightly coupled Data growth drives distributed MPP architectures but processing and data storage still tightly coupled Further data growth drives distributed file system architecture. Processing and data storage co- located but loosely coupled 1970s 1980s 2000s 2010s
- 6. The Big DataEcosystem Co-located compute and storage for big data workloads § More defined and loosely coupled compute layer compared with relational databases § But compute / data processing still runs on the same node as where the data is stored. MapReduce runs on HDFS across the cluster § Compute layer and storage layer must be scaled out by the same factor
- 7. The Big DataEcosystem Explodes Moving from tightly integrated to loosely integrated architectures STORAGE COMPUTE
- 8. Why independently scalecompute and storage for data-driven applications? Flexible compute scaling based on application demands Flexible storage scaling based on data growth patterns Compute is CPU bound Storage is I/O bound
- 9. Why independently scalecompute and storage for data-driven applications? Flexibility of using hardware / instances that match the workload CPU / GPU + IO + Memory S3 Leverage cheaper and newer storage like object stores for big data / AI workloads Orchestrate & automate compute for greater operational efficiency Protect & control your data on premises and leverage public cloud for compute
- 10. The challenges ofindependent scaling for data-driven workloads Data Locality Data Accessibility Data Abstraction Data is no more local to compute and workload processing time will increase particularly in hybrid cloud deployments Data is in multiple storage systems in multiple locations. Highly complex when all compute frameworks talk to all storage systems Data can still only be accessed using the specific storage system APIs
- 11. Java File APIHDFS Interface S3 Interface REST APIPOSIX Interface HDFS Driver Swift Driver S3 Driver NFS Driver
- 12.
- 13. Unified Namespace Bring all filesinto a single interface Interact with data using any API Accelerate & tier data transparently API Translation Intelligent Multi-tiering Key Innovations of theVirtual Unified File System
- 14. Data Abstraction viaUnified Namespace Enables effective data management across different Under Store - Uses Mounting withTransparent Naming
- 15. Unified Namespace: GlobalData Accessibility Transparent access to understorage makes all enterprise data available locally SUPPORTS • HDFS • NFS • OpenStack • Ceph • Amazon S3 • Azure • Google Cloud IT OPS FRIENDLY • Storage mounted into Alluxio by central IT • Security in Alluxio mirrors source data • Authentication through LDAP/AD • Wireline encryption HDFS #1 Object Store NFS HDFS #2 Interacting with data in Alluxio – Unified Namespace
- 16. Data Accessibility viaServer-side API Translation Convert from Client-side Interface to native Storage Interface Java File API HDFS Interface S3 Interface REST APIFUSE Interface HDFS Driver Swift DriverS3 Driver NFS Driver
- 17. Interacting with datain Alluxio – variety of APIs Spark Hadoop POSIX Java Application have great flexibility to read / write data with many options > rdd = sc.textFile(“alluxio://localhost:19998/myInput”) $ hadoop fs -cat alluxio://localhost:19998/myInput $ cat /mnt/alluxio/myInput FileSystem fs = FileSystem.Factory.get(); FileInStream in = fs.openFile(new AlluxioURI("/myInput"));
- 18. Data Locality viaIntelligent Multi-tiering Local performance from remote data using multi-tier storage Hot Warm Cold RAM SSD HDD Read & Write Buffering Transparent to App Policies for pinning, promotion/demotion,TTL
- 19. Interacting with datain Alluxio – flexible app patterns Reading Data • Data not in Alluxio (i.e. first time, or no cache) • Data on same node as client • Data on different node from client Writing Data • Write only to Alluxio • Write only to Under Store • Write synchronously to Alluxio and Under Store • Write to Alluxio and asynchronously write to Under Store • Write to Alluxio and replicate to N other workers • Write to Alluxio and async write to multiple Under stores Application have great flexibility to read / write data with many options
- 20. Interacting with datain Alluxio – data management Data Management • Pinning • Prefetch/free • Cross mount points cp/mv • TTL Application have great flexibility to read / write data with many options
- 21. Demo: Spark +Alluxio + S3
- 22. Deployment Approaches Spark Alluxio Storage Co-locate AlluxioWorkers with Spark for optimal I/O performance Any Cloud Same instance / container Spark Alluxio Storage Deploy Alluxio as standalone cluster between Spark and Storage Any Cloud Same data center / region Presto
- 23. Alluxio Master Zookeeper / RAFT Standby Master Under Store1 Under Store 2 WANAlluxio Client Application Object Store Alluxio Client Application Alluxio Worker RAM / SSD / HDD Alluxio Worker RAM / SSD / HDD Alluxio Reference Architecture
- 24. Read data inAlluxio, on same node as client 24 Alluxio Worker RAM / SSD / HDD Memory Speed Read of Data Application Alluxio Client Alluxio Master
- 25. Read data notin Alluxio 25 RAM / SSD / HDD Network / Disk Speed Read of Data Application Alluxio Client Alluxio Master Alluxio WorkerUnder Store
- 26. Write data onlyto Alluxio on same node as client 26 Alluxio Worker RAM / SSD / HDD Memory Speed Write of Data Application Alluxio Client Alluxio Master
- 27. Write data toAlluxio and Under Store synchronously 27 RAM / SSD / HDD Network / Disk Speed Write of Data Application Alluxio Client Alluxio Master Alluxio Worker Under Store
- 28. Real world Usecases
- 29. § Huya: 1300s(100% Hadoop nodes) § Sogou: 1000s (60% Hadoop nodes) § Momo: 850 nodes § JD: 600 nodes § Tencent: 400 nodes § Huawei Cloud: 150 nodes § China Unicom: 100 nodes § … Truly independent scaling of the data stack Alluxio Production Deployments
- 30. China Unicom Challenge Desired acentral view of business data across multiple systems for big data workloads Solution Alluxio integrates data across multiple storage system to be accessed by Spark in a hybrid environment Impact Significantly faster workloads and faster innovation
- 31. Machine Learning CaseStudy Challenge – Gain end to end view of business with large volume of data Queries were slow / not interactive, resulting in operational inefficiency Solution – ETL Data from Teradata to Alluxio Impact – Faster Time to Market – “Now we don’t have to work Sundays” Use Case: http://bit.ly/2oMx95W SPARK TERADATA SPARK TERADATA
- 32. Analytics Use Case– Top Retailer Challenge – Bottleneck in Trend Analysis of mission critical daily sales and inventory management Queries were slow / not interactive, resulting in operational inefficiency Solution – With Alluxio, data queries are 10X faster Impact – Higher operational efficiency Use case: http://bit.ly/2ook8Nh SPARK HDFS SPARK HDFS
- 33. Truly independent scalingof the data stack Coming Online Events: POSIX API & 2.0 Preview https://go.alluxio.org/OH-Running-Tensorflow-Alluxio-S3 https://go.alluxio.com/webinar-alluxio-preview-release
- 34. Incredible Open SourceMomentum with growing community 900+ contributors & growing 3900+ Git Stars Apache 2.0 Licensed Hundreds of thousands of downloads https://www.alluxio.org/slack