Download as PDF, PPTX
Uploaded byAlluxio, Inc.
Decoupling Compute and Storage for Data Workloads
The document discusses the challenges of scaling compute and storage in data processing workloads, particularly in relation to Hadoop at DBS. It proposes decoupling compute and storage to improve flexibility, performance, and cost-effectiveness through technologies such as Alluxio and object stores. Current implementation status includes development with virtual machines and performance benchmarks using Cloudera and S3-compatible environments.
More Related Content
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
Decoupling Compute and Storage for Data Workloads
- 1. Decoupling compute and storage fordata workloads Carlos Queiroz
- 2. Data processing workloadsat DBS Hadoop introduced in 2015 Business and Regulatory Reporting DataWareHouse replacement? Analytics datanode JVM DataNode1 datanode JVM DataNode2 datanode JVM DataNodeX … namenode JVM NameNode namenode JVM NameNode ETL Batch Bare-Metal Data Locality HDFS on local disks Enterprise transactions Logs mainframe H D F S ETL Processing Data Science H D F S User ETL ETL
- 3.
- 4. Current model • Hardto scale • Scale Compute AND Storage • It is not flexible • Costs Bare-Metal Data Locality HDFS on local disks
- 5. Also in 2015 EMCand Adobe bringing HDaaS https://www.brighttalk.com/webcast/1744/156173
- 6. Decoupling compute andstorage Bare-Metal Data Locality HDFS on local disks Containers and VMs Separate Compute and Storage Shared Storage Data as a Service Agility and cost savings Faster time to foresights Traditional Assumptions A New Approach Benefits and Value https://www.bluedata.com/blog/2015/12/separating-hadoop-compute-and-storage/Adapted from
- 7. Fast Forward to2017 Re-engineering the data platform StorageCompute DataIngestion Decisionsupport Object store In-memory Filesystem Compute engine I Compute Engine II Compute Engine III Compute Engine IV …
- 8. Fast Forward to2017 Storage Object store In-memory Filesystem Compute Compute engine I Compute Engine II Compute Compute engine I Compute Compute engine I Compute Engine II Compute Engine X Multi-tenancy Different SLAs Different Engines Different Cluster sizes
- 9.
- 10. In-memory filesystem • Appsonly talk to Alluxio • Simple Add/Remove • No App Changes • Highest performance in Memory
- 11. Alluxio - Serverside API translation HDFS Interface S3A Interface S3 compatible S3 compatible software
- 12.
- 13. Current implementation status •Development environment with 50 VMs • Running benchmarks for performance evaluation • Cloudera 5.13.x • S3 compatible object store vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB vCPU: 12 RAM: 128 Disk: 400GB
- 14.