Alluxio, Inc., profile picture
Uploaded byAlluxio, Inc.
876 views

Accelerating Spark Workloads in a Mesos Environment with Alluxio

Gene Pang presents on accelerating Spark workloads in a Mesos environment using Alluxio. Alluxio provides a unified memory-speed storage layer across multiple storage systems and compute frameworks like Spark. Using Alluxio with Spark on Mesos allows sharing data in memory between processes for faster performance compared to keeping data on disk. Alluxio also provides data resilience during process crashes by keeping data blocks cached in memory. The presentation demonstrates setting up Alluxio with Spark on Mesos and shows an 8x performance improvement.

Embed presentation

Downloaded 11 times
1 Accelerating Spark Workloads in a Mesos Environment with Alluxio Gene Pang, Software Engineer, Alluxio, Inc. * ©2017 Alluxio, Inc. All Rights Reserved
About Me Gene Pang Software Engineer @ Alluxio, Inc. Alluxio Open Source PMC Member Ph.D. from AMPLab @ UC Berkeley Worked at Google before UC Berkeley Twitter: @unityxx Github: @gpang ©2017 Alluxio, Inc. All Rights Reserved 2
Outline Alluxio Overview Alluxio + Spark + Mesos Use Cases Using Spark with Alluxio on Mesos Deployment with Mesos Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 3
Data Ecosystem Yesterday 4* ©2017 Alluxio, Inc. All Rights Reserved • One Compute Framework • Single Storage System • Co-located
Data Ecosystem Today 5* ©2017 Alluxio, Inc. All Rights Reserved … • Many Compute Frameworks • Multiple Storage Systems • Most not co-located …
Data Ecosystem Issues 6* ©2017 Alluxio, Inc. All Rights Reserved • Each application manage multiple data sources • Add/Removing data sources require application changes • Storage optimizations requires application change • Lower performance due to lack of locality … …
Data Ecosystem with Alluxio 7* ©2017 Alluxio, Inc. All Rights Reserved • Apps only talk to Alluxio • Simple Add/Remove • No App Changes • Memory Performance … …
Next Gen Analytics with Alluxio 8* ©2017 Alluxio, Inc. All Rights Reserved ✓ Big Data/IoT ✓ AI/ML ✓ Deep Learning ✓ Cloud Migration ✓ Multi Platform ✓ Autonomous … … Native File System Hadoop Compatible File System Native Key-Value Interface Fuse Compatible File System HDFS Interface Amazon S3 Interface Swift Interface GlusterFS Interface Apps, Data & Storage at Memory Speed
Enabling Next Gen Analytics Unify your Data 9 1 Architecture Flexibility2 Improved I/O Performance3 * ©2017 Alluxio, Inc. All Rights Reserved
Fastest Growing Big Data Open Source Project 10* ©2017 Alluxio, Inc. All Rights Reserved • Fastest Growing open- source project in the big data ecosystem • Running world’s largest production clusters • 600+ Contributors from 100+ organizations
Outline Alluxio Overview Alluxio + Spark + Mesos Use Cases Using Spark with Alluxio on Mesos Deployment with Mesos Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 11
Big Data Case Study – Challenge – Gain end to end view of business with large volume of data for $5B Travel Site Queries were slow / not interactive, resulting in operational inefficiency SPARK HDFS Solution – With Alluxio, 300x improvement in performance Impact – Increased revenue from immediate response to user behavior Use case: http://bit.ly/2pDJdrq CEPH HDFS CEPH FLINK SPARK FLINK ©2017 Alluxio, Inc. All Rights Reserved 12 MESOS
Machine Learning Case Study – 136/12/17 ©2017 Alluxio, Inc. All Rights Reserved Challenge – Disparate Data both on-prem and Cloud. Heterogeneous types of data. Scaling of Exabyte size data. Slow due to disk based approach. SPARK HDFS SPARK MINIO Solution – Using Alluxio to prevent I/O bottlenecks Impact – Orders of magnitude higher performance than before. http://bit.ly/2p18ds3 MESOS
Outline Alluxio Overview Alluxio + Spark + Mesos Use Cases Using Spark with Alluxio on Mesos Deployment with Mesos Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 14
Sharing Data via Memory Storage Engine & Execution Engine Same Process • Two copies of data in memory – double the memory used • Inter-process Sharing Slowed Down by Network / Disk I/O ©2017 Alluxio, Inc. All Rights Reserved 15 Mesos Spark Compute Spark Storage block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 Spark Compute Spark Storage block 1 block 3
Sharing Data via Memory Storage Engine & Execution Engine Different process • Half the memory used • Inter-process Sharing Happens at Memory Speed Spark Compute Spark Storage HDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Alluxio block 1 block 3 block 4 Spark Compute Spark Storage ©2017 Alluxio, Inc. All Rights Reserved 16 Mesos
Data Resilience During Crash Spark Compute Spark Storage block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 Storage Engine & Execution Engine Same Process ©2017 Alluxio, Inc. All Rights Reserved 17 Mesos
Data Resilience During Crash CRASH Spark Storage block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 4 block 2 • Process Crash Requires Network and/or Disk I/O to Re-read Data Storage Engine & Execution Engine Same Process ©2017 Alluxio, Inc. All Rights Reserved 18 Mesos
Data Resilience During Crash CRASH HDFS / Amazon S3 block 1 block 3 block 2 block 4 Storage Engine & Execution Engine Same Process • Process Crash Requires Network and/or Disk I/O to Re-read Data ©2017 Alluxio, Inc. All Rights Reserved 19 Mesos
Data Resilience During Crash Spark Compute Spark Storage HDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Alluxio block 1 block 3 block 4 Storage Engine & Execution Engine Different process ©2017 Alluxio, Inc. All Rights Reserved 20 Mesos
Data Resilience During Crash Process Crash - Data is Re-read at Memory SpeedHDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Alluxio block 1 block 3 block 4 CRASH Storage Engine & Execution Engine Different process ©2017 Alluxio, Inc. All Rights Reserved 21 Mesos
Alluxio Architecture ©2017 Alluxio, Inc. All Rights Reserved 22 Application AlluxioClient Alluxio Master Alluxio Worker Alluxio Worker … Storage Storage …
Alluxio Client ©2017 Alluxio, Inc. All Rights Reserved 23 Applications interact with Alluxio via the Alluxio client ● Native Alluxio Filesystem Client • Alluxio specific operations like [un]pin, [un]mount, [un]set TTL ● HDFS-Compatible Filesystem Client • No code change necessary ● S3 API
Alluxio Master ©2017 Alluxio, Inc. All Rights Reserved 24 Master is responsible for managing metadata ● Filesystem namespace metadata ● Blocks / workers metadata Primary master writes journal for durable operations ● Secondary masters replay journal entries
Alluxio Worker ©2017 Alluxio, Inc. All Rights Reserved 25 Worker is responsible for managing block data Worker stores block data on various storage media ● HDD, SSD, Memory Reads and writes data to underlying storage systems
Outline Alluxio Overview Alluxio + Spark + Mesos Use Cases Using Spark with Alluxio on Mesos Deployment with Mesos Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 26
Alluxio on DC/OS ©2017 Alluxio, Inc. All Rights Reserved 27
Alluxio on DC/OS ©2017 Alluxio, Inc. All Rights Reserved 28 Alluxio brings A unified view of data across disparate storage systems High performance & predictable SLA for analytics workloads DC/OS makes provisioning infrastructure easy Automates provisioning, management & elastic scaling Benefits include: Faster analytics with Spark and other frameworks Process data from hybrid cloud storage systems (HDFS, S3, etc)
Outline Alluxio Overview Alluxio + Spark + Mesos Use Cases Using Spark with Alluxio on Mesos Deployment with Mesos Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 29
Demo Environment Spark Alluxio ©2017 Alluxio, Inc. All Rights Reserved 30 SPARK MESOS
Demo Setup Alluxio 1.5.0 DC/OS 1.9.4 Spark 2.0.2 Amazon EC2 (m3.xlarge) ©2017 Alluxio, Inc. All Rights Reserved 31
Results ©2017 Alluxio, Inc. All Rights Reserved 32 8x  improvement  
Conclusion Easy to use Alluxio with Spark in a Mesos environment Predictable and improved performance Easily connect to various storage systems ©2017 Alluxio, Inc. All Rights Reserved 33
Thank you! Gene Pang Software Engineer gene@alluxio.com 34 Twitter.com/alluxio Linkedin.com/alluxio Website www.alluxio.com E-mail info@alluxio.com @ Social Media * ©2017 Alluxio, Inc. All Rights Reserved

More Related Content

PDF
Best Practices for Using Alluxio with Spark
byAlluxio, Inc.
 
PDF
Alluxio Mesos Meetup - SMACK to SMAACK
byAlluxio, Inc.
 
PDF
Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017
byAlluxio, Inc.
 
PDF
Alluxio: Unify Data at Memory Speed at Strata and Hadoop World San Jose 2017
byAlluxio, Inc.
 
PDF
Introduction to Alluxio (formerly Tachyon) and how it brings up to 300x perfo...
byAlluxio, Inc.
 
PDF
Spark Pipelines in the Cloud with Alluxio
byAlluxio, Inc.
 
PDF
Unify Data at Memory Speed by Haoyuan Li - VAULT Conference 2017
byAlluxio, Inc.
 
PDF
Best Practices for Using Alluxio with Spark
byAlluxio, Inc.
 
Best Practices for Using Alluxio with Spark
byAlluxio, Inc.
 
Alluxio Mesos Meetup - SMACK to SMAACK
byAlluxio, Inc.
 
Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017
byAlluxio, Inc.
 
Alluxio: Unify Data at Memory Speed at Strata and Hadoop World San Jose 2017
byAlluxio, Inc.
 
Introduction to Alluxio (formerly Tachyon) and how it brings up to 300x perfo...
byAlluxio, Inc.
 
Spark Pipelines in the Cloud with Alluxio
byAlluxio, Inc.
 
Unify Data at Memory Speed by Haoyuan Li - VAULT Conference 2017
byAlluxio, Inc.
 
Best Practices for Using Alluxio with Spark
byAlluxio, Inc.
 

What's hot

PDF
Best Practices for Using Alluxio with Spark
byAlluxio, Inc.
 
PDF
Enable Fast Big Data Analytics on Ceph with Alluxio at Ceph Days 2017
byAlluxio, Inc.
 
PDF
ALLUXIO (formerly Tachyon): Unify Data at Memory Speed - Effective using Spar...
byAlluxio, Inc.
 
PDF
Alluxio (Formerly Tachyon): Unify Data At Memory Speed at Global Big Data Con...
byAlluxio, Inc.
 
PDF
Alluxio Use Cases at Strata+Hadoop World Beijing 2016
byAlluxio, Inc.
 
PDF
Open Source Memory Speed Virtual Distributed Storage
byAlluxio, Inc.
 
PDF
Flexible and Fast Storage for Deep Learning with Alluxio
byAlluxio, Inc.
 
PDF
Alluxio (formerly Tachyon): Open Source Memory Speed Virtual Distributed Storage
byAlluxio, Inc.
 
PPTX
Running Solr in the Cloud at Memory Speed with Alluxio
bythelabdude
 
PDF
Alluxio: Unify Data at Memory Speed; 2016-11-18
byAlluxio, Inc.
 
PDF
Securely Enhancing Data Access in Hybrid Cloud with Alluxio
byAlluxio, Inc.
 
PDF
Speeding Up Spark Performance using Alluxio at China Unicom
byAlluxio, Inc.
 
PDF
Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...
byAlluxio, Inc.
 
PDF
Building a high-performance data lake analytics engine at Alibaba Cloud with ...
byAlluxio, Inc.
 
PDF
Spark Summit EU talk by Jiri Simsa
bySpark Summit
 
PPTX
Alluxio Presentation at Strata San Jose 2016
byJiří Šimša
 
PDF
Alluxio Keynote at Strata+Hadoop World Beijing 2016
byAlluxio, Inc.
 
PDF
Alluxio (formerly Tachyon): The Journey thus far and the Road Ahead
byAlluxio, Inc.
 
PDF
Alluxio Presentation at AMPLab Summer Retreat 2016
byAlluxio, Inc.
 
PDF
Alluxio-FUSE as a data access layer for Dask
byAlluxio, Inc.
 
Best Practices for Using Alluxio with Spark
byAlluxio, Inc.
 
Enable Fast Big Data Analytics on Ceph with Alluxio at Ceph Days 2017
byAlluxio, Inc.
 
ALLUXIO (formerly Tachyon): Unify Data at Memory Speed - Effective using Spar...
byAlluxio, Inc.
 
Alluxio (Formerly Tachyon): Unify Data At Memory Speed at Global Big Data Con...
byAlluxio, Inc.
 
Alluxio Use Cases at Strata+Hadoop World Beijing 2016
byAlluxio, Inc.
 
Open Source Memory Speed Virtual Distributed Storage
byAlluxio, Inc.
 
Flexible and Fast Storage for Deep Learning with Alluxio
byAlluxio, Inc.
 
Alluxio (formerly Tachyon): Open Source Memory Speed Virtual Distributed Storage
byAlluxio, Inc.
 
Running Solr in the Cloud at Memory Speed with Alluxio
bythelabdude
 
Alluxio: Unify Data at Memory Speed; 2016-11-18
byAlluxio, Inc.
 
Securely Enhancing Data Access in Hybrid Cloud with Alluxio
byAlluxio, Inc.
 
Speeding Up Spark Performance using Alluxio at China Unicom
byAlluxio, Inc.
 
Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...
byAlluxio, Inc.
 
Building a high-performance data lake analytics engine at Alibaba Cloud with ...
byAlluxio, Inc.
 
Spark Summit EU talk by Jiri Simsa
bySpark Summit
 
Alluxio Presentation at Strata San Jose 2016
byJiří Šimša
 
Alluxio Keynote at Strata+Hadoop World Beijing 2016
byAlluxio, Inc.
 
Alluxio (formerly Tachyon): The Journey thus far and the Road Ahead
byAlluxio, Inc.
 
Alluxio Presentation at AMPLab Summer Retreat 2016
byAlluxio, Inc.
 
Alluxio-FUSE as a data access layer for Dask
byAlluxio, Inc.
 

Similar to Accelerating Spark Workloads in a Mesos Environment with Alluxio

PDF
Accelerating Spark Workloads in an Apache Mesos Environment with Alluxio
byAlluxio, Inc.
 
PDF
Best Practices for Using Alluxio with Apache Spark with Gene Pang
bySpark Summit
 
PPTX
Spark Pipelines in the Cloud with Alluxio by Bin Fan
byData Con LA
 
PDF
Best Practices for Using Alluxio with Apache Spark with Cheng Chang and Haoyu...
byDatabricks
 
PDF
Best Practice in Accelerating Data Applications with Spark+Alluxio
byAlluxio, Inc.
 
PDF
Spark Summit EU talk by Jiri Simsa
byAlluxio, Inc.
 
PDF
Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
byAlluxio, Inc.
 
PDF
Alluxio @ Uber Seattle Meetup
byAlluxio, Inc.
 
PDF
Getting Started with Alluxio + Spark + S3
byAlluxio, Inc.
 
PDF
Accelerate Spark Workloads on S3
byAlluxio, Inc.
 
PDF
Improving Memory Utilization of Spark Jobs Using Alluxio
byAlluxio, Inc.
 
PPTX
Alluxio: Unify Data at Memory Speed
byAlluxio, Inc.
 
PDF
Accelerating Spark with Kubernetes
byAlluxio, Inc.
 
PDF
Achieving compute and storage independence for data-driven workloads
byAlluxio, Inc.
 
PDF
Alluxio 2.0 Deep Dive – Simplifying data access for cloud workloads
byAlluxio, Inc.
 
PDF
Simplified Data Preparation for Machine Learning in Hybrid and Multi Clouds
byAlluxio, Inc.
 
PDF
The Architecture of Decoupling Compute and Storage with Alluxio
byAlluxio, Inc.
 
PDF
Effective Spark with Alluxio: Spark Summit East talk by Gene Pang and Haoyuan...
bySpark Summit
 
PDF
Achieving Separation of Compute and Storage in a Cloud World
byAlluxio, Inc.
 
PDF
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
byAlluxio, Inc.
 
Accelerating Spark Workloads in an Apache Mesos Environment with Alluxio
byAlluxio, Inc.
 
Best Practices for Using Alluxio with Apache Spark with Gene Pang
bySpark Summit
 
Spark Pipelines in the Cloud with Alluxio by Bin Fan
byData Con LA
 
Best Practices for Using Alluxio with Apache Spark with Cheng Chang and Haoyu...
byDatabricks
 
Best Practice in Accelerating Data Applications with Spark+Alluxio
byAlluxio, Inc.
 
Spark Summit EU talk by Jiri Simsa
byAlluxio, Inc.
 
Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
byAlluxio, Inc.
 
Alluxio @ Uber Seattle Meetup
byAlluxio, Inc.
 
Getting Started with Alluxio + Spark + S3
byAlluxio, Inc.
 
Accelerate Spark Workloads on S3
byAlluxio, Inc.
 
Improving Memory Utilization of Spark Jobs Using Alluxio
byAlluxio, Inc.
 
Alluxio: Unify Data at Memory Speed
byAlluxio, Inc.
 
Accelerating Spark with Kubernetes
byAlluxio, Inc.
 
Achieving compute and storage independence for data-driven workloads
byAlluxio, Inc.
 
Alluxio 2.0 Deep Dive – Simplifying data access for cloud workloads
byAlluxio, Inc.
 
Simplified Data Preparation for Machine Learning in Hybrid and Multi Clouds
byAlluxio, Inc.
 
The Architecture of Decoupling Compute and Storage with Alluxio
byAlluxio, Inc.
 
Effective Spark with Alluxio: Spark Summit East talk by Gene Pang and Haoyuan...
bySpark Summit
 
Achieving Separation of Compute and Storage in a Cloud World
byAlluxio, Inc.
 
Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio
byAlluxio, Inc.
 

More from Alluxio, Inc.

PDF
AI/ML Infra Meetup | Open Source Michelangelo: Uber's Predictive to Generativ...
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | Unlock the Future of Generative AI: TorchTitan's Latest ...
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | Bringing Data to GPUs Anywhere + Get Low-Latency on Obje...
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | SkyPilot: Open-source System to Scale AI across Clusters...
byAlluxio, Inc.
 
PDF
Alluxio Webinar | Alluxio + S3 A Tiered Architecture for Latency-Critical, Se...
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | AI at scale Architecting Scalable, Deployable and Resili...
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | Achieving Double-Digit Millisecond Offline Feature Store...
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | Alluxio + S3 A Tiered Architecture for Latency-Critical,...
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | Beyond S3's Basics: Architecting for AI-Native Data Access
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | LLM Agents and Implementation Challenges
byAlluxio, Inc.
 
PDF
Product Update: Alluxio AI 3.7 Now with Sub-Millisecond Latency
byAlluxio, Inc.
 
PDF
Introduction to Apache Iceberg™ & Tableflow
byAlluxio, Inc.
 
PDF
Optimizing Tiered Storage for Low-Latency Real-Time Analytics at AI Scale
byAlluxio, Inc.
 
PDF
Meet in the Middle: Solving the Low-Latency Challenge for Agentic AI
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | From Data Preparation to Inference: How Alluxio Speeds U...
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | Best Practice for LLM Serving in the Cloud
byAlluxio, Inc.
 
PDF
Meet You in the Middle: 1000x Performance for Parquet Queries on PB-Scale Dat...
byAlluxio, Inc.
 
PDF
How Coupang Leverages Distributed Cache to Accelerate ML Model Training
byAlluxio, Inc.
 
PDF
Alluxio Webinar | Inside Deepseek 3FS: A Deep Dive into AI-Optimized Distribu...
byAlluxio, Inc.
 
PDF
AI/ML Infra Meetup | Building Production Platform for Large-Scale Recommendat...
byAlluxio, Inc.
 
AI/ML Infra Meetup | Open Source Michelangelo: Uber's Predictive to Generativ...
byAlluxio, Inc.
 
AI/ML Infra Meetup | Unlock the Future of Generative AI: TorchTitan's Latest ...
byAlluxio, Inc.
 
AI/ML Infra Meetup | Bringing Data to GPUs Anywhere + Get Low-Latency on Obje...
byAlluxio, Inc.
 
AI/ML Infra Meetup | SkyPilot: Open-source System to Scale AI across Clusters...
byAlluxio, Inc.
 
Alluxio Webinar | Alluxio + S3 A Tiered Architecture for Latency-Critical, Se...
byAlluxio, Inc.
 
AI/ML Infra Meetup | AI at scale Architecting Scalable, Deployable and Resili...
byAlluxio, Inc.
 
AI/ML Infra Meetup | Achieving Double-Digit Millisecond Offline Feature Store...
byAlluxio, Inc.
 
AI/ML Infra Meetup | Alluxio + S3 A Tiered Architecture for Latency-Critical,...
byAlluxio, Inc.
 
AI/ML Infra Meetup | Beyond S3's Basics: Architecting for AI-Native Data Access
byAlluxio, Inc.
 
AI/ML Infra Meetup | LLM Agents and Implementation Challenges
byAlluxio, Inc.
 
Product Update: Alluxio AI 3.7 Now with Sub-Millisecond Latency
byAlluxio, Inc.
 
Introduction to Apache Iceberg™ & Tableflow
byAlluxio, Inc.
 
Optimizing Tiered Storage for Low-Latency Real-Time Analytics at AI Scale
byAlluxio, Inc.
 
Meet in the Middle: Solving the Low-Latency Challenge for Agentic AI
byAlluxio, Inc.
 
AI/ML Infra Meetup | From Data Preparation to Inference: How Alluxio Speeds U...
byAlluxio, Inc.
 
AI/ML Infra Meetup | Best Practice for LLM Serving in the Cloud
byAlluxio, Inc.
 
Meet You in the Middle: 1000x Performance for Parquet Queries on PB-Scale Dat...
byAlluxio, Inc.
 
How Coupang Leverages Distributed Cache to Accelerate ML Model Training
byAlluxio, Inc.
 
Alluxio Webinar | Inside Deepseek 3FS: A Deep Dive into AI-Optimized Distribu...
byAlluxio, Inc.
 
AI/ML Infra Meetup | Building Production Platform for Large-Scale Recommendat...
byAlluxio, Inc.
 

Recently uploaded

PDF
Accelerating Responsible AI Adoption in Public Sector and Private Organizations.
byYasir Naveed Riaz
 
PDF
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
PDF
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
PDF
Decoding the DNA: The Digital Networks Act, the Open Internet, and IP interco...
byCSUC - Consorci de Serveis Universitaris de Catalunya
 
PPTX
From Backup to Resilience: How MSPs Are Preparing for 2026
byMSP360
 
PDF
Eredità digitale sugli smartphone: cosa resta di noi nei dispositivi mobili
bySpeck&Tech
 
PDF
Day 5 - Red Team + Blue Team in the Cloud - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
PDF
Day 3 - Data and Application Security - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
PPTX
cybercrime in Information security .pptx
byismaeelsahib032
 
PDF
GPUS and How to Program Them by Manya Bansal
byScyllaDB
 
PPTX
Cybersecurity Best Practices - Step by Step guidelines
byYasir Naveed Riaz
 
PDF
Security Technologys: Access Control, Firewall, VPN
byShielaLasala
 
PPTX
Cloud-and-AI-Platform-FY26-Partner-Playbook.pptx
byiuiuiuiu1
 
PDF
Digit Expo 2025 - EICC Edinburgh 27th November
byRay Bugg
 
PDF
Day 2 - Network Security ~ 2nd Sight Lab ~ Cloud Security Class ~ 2020
by2nd Sight Lab
 
PDF
Safeguarding AI-Based Financial Infrastructure
byYasir Naveed Riaz
 
PDF
Zero Trust & Defense-in-Depth: The Future of Critical Infrastructure Security
byYasir Naveed Riaz
 
PDF
Making Sense of Raster: From Bit Depth to Better Workflows
bySafe Software
 
DOCX
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 
PDF
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
Accelerating Responsible AI Adoption in Public Sector and Private Organizations.
byYasir Naveed Riaz
 
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
Decoding the DNA: The Digital Networks Act, the Open Internet, and IP interco...
byCSUC - Consorci de Serveis Universitaris de Catalunya
 
From Backup to Resilience: How MSPs Are Preparing for 2026
byMSP360
 
Eredità digitale sugli smartphone: cosa resta di noi nei dispositivi mobili
bySpeck&Tech
 
Day 5 - Red Team + Blue Team in the Cloud - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
Day 3 - Data and Application Security - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
cybercrime in Information security .pptx
byismaeelsahib032
 
GPUS and How to Program Them by Manya Bansal
byScyllaDB
 
Cybersecurity Best Practices - Step by Step guidelines
byYasir Naveed Riaz
 
Security Technologys: Access Control, Firewall, VPN
byShielaLasala
 
Cloud-and-AI-Platform-FY26-Partner-Playbook.pptx
byiuiuiuiu1
 
Digit Expo 2025 - EICC Edinburgh 27th November
byRay Bugg
 
Day 2 - Network Security ~ 2nd Sight Lab ~ Cloud Security Class ~ 2020
by2nd Sight Lab
 
Safeguarding AI-Based Financial Infrastructure
byYasir Naveed Riaz
 
Zero Trust & Defense-in-Depth: The Future of Critical Infrastructure Security
byYasir Naveed Riaz
 
Making Sense of Raster: From Bit Depth to Better Workflows
bySafe Software
 
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 

Accelerating Spark Workloads in a Mesos Environment with Alluxio

  • 1.
    1 Accelerating Spark Workloadsin a Mesos Environment with Alluxio Gene Pang, Software Engineer, Alluxio, Inc. * ©2017 Alluxio, Inc. All Rights Reserved
  • 2.
    About Me Gene Pang SoftwareEngineer @ Alluxio, Inc. Alluxio Open Source PMC Member Ph.D. from AMPLab @ UC Berkeley Worked at Google before UC Berkeley Twitter: @unityxx Github: @gpang ©2017 Alluxio, Inc. All Rights Reserved 2
  • 3.
    Outline Alluxio Overview Alluxio +Spark + Mesos Use Cases Using Spark with Alluxio on Mesos Deployment with Mesos Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 3
  • 4.
    Data Ecosystem Yesterday 4*©2017 Alluxio, Inc. All Rights Reserved • One Compute Framework • Single Storage System • Co-located
  • 5.
    Data Ecosystem Today 5*©2017 Alluxio, Inc. All Rights Reserved … • Many Compute Frameworks • Multiple Storage Systems • Most not co-located …
  • 6.
    Data Ecosystem Issues 6*©2017 Alluxio, Inc. All Rights Reserved • Each application manage multiple data sources • Add/Removing data sources require application changes • Storage optimizations requires application change • Lower performance due to lack of locality … …
  • 7.
    Data Ecosystem withAlluxio 7* ©2017 Alluxio, Inc. All Rights Reserved • Apps only talk to Alluxio • Simple Add/Remove • No App Changes • Memory Performance … …
  • 8.
    Next Gen Analyticswith Alluxio 8* ©2017 Alluxio, Inc. All Rights Reserved ✓ Big Data/IoT ✓ AI/ML ✓ Deep Learning ✓ Cloud Migration ✓ Multi Platform ✓ Autonomous … … Native File System Hadoop Compatible File System Native Key-Value Interface Fuse Compatible File System HDFS Interface Amazon S3 Interface Swift Interface GlusterFS Interface Apps, Data & Storage at Memory Speed
  • 9.
    Enabling Next GenAnalytics Unify your Data 9 1 Architecture Flexibility2 Improved I/O Performance3 * ©2017 Alluxio, Inc. All Rights Reserved
  • 10.
    Fastest Growing BigData Open Source Project 10* ©2017 Alluxio, Inc. All Rights Reserved • Fastest Growing open- source project in the big data ecosystem • Running world’s largest production clusters • 600+ Contributors from 100+ organizations
  • 11.
    Outline Alluxio Overview Alluxio +Spark + Mesos Use Cases Using Spark with Alluxio on Mesos Deployment with Mesos Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 11
  • 12.
    Big Data CaseStudy – Challenge – Gain end to end view of business with large volume of data for $5B Travel Site Queries were slow / not interactive, resulting in operational inefficiency SPARK HDFS Solution – With Alluxio, 300x improvement in performance Impact – Increased revenue from immediate response to user behavior Use case: http://bit.ly/2pDJdrq CEPH HDFS CEPH FLINK SPARK FLINK ©2017 Alluxio, Inc. All Rights Reserved 12 MESOS
  • 13.
    Machine Learning CaseStudy – 136/12/17 ©2017 Alluxio, Inc. All Rights Reserved Challenge – Disparate Data both on-prem and Cloud. Heterogeneous types of data. Scaling of Exabyte size data. Slow due to disk based approach. SPARK HDFS SPARK MINIO Solution – Using Alluxio to prevent I/O bottlenecks Impact – Orders of magnitude higher performance than before. http://bit.ly/2p18ds3 MESOS
  • 14.
    Outline Alluxio Overview Alluxio +Spark + Mesos Use Cases Using Spark with Alluxio on Mesos Deployment with Mesos Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 14
  • 15.
    Sharing Data viaMemory Storage Engine & Execution Engine Same Process • Two copies of data in memory – double the memory used • Inter-process Sharing Slowed Down by Network / Disk I/O ©2017 Alluxio, Inc. All Rights Reserved 15 Mesos Spark Compute Spark Storage block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 Spark Compute Spark Storage block 1 block 3
  • 16.
    Sharing Data viaMemory Storage Engine & Execution Engine Different process • Half the memory used • Inter-process Sharing Happens at Memory Speed Spark Compute Spark Storage HDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Alluxio block 1 block 3 block 4 Spark Compute Spark Storage ©2017 Alluxio, Inc. All Rights Reserved 16 Mesos
  • 17.
    Data Resilience DuringCrash Spark Compute Spark Storage block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 Storage Engine & Execution Engine Same Process ©2017 Alluxio, Inc. All Rights Reserved 17 Mesos
  • 18.
    Data Resilience DuringCrash CRASH Spark Storage block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 4 block 2 • Process Crash Requires Network and/or Disk I/O to Re-read Data Storage Engine & Execution Engine Same Process ©2017 Alluxio, Inc. All Rights Reserved 18 Mesos
  • 19.
    Data Resilience DuringCrash CRASH HDFS / Amazon S3 block 1 block 3 block 2 block 4 Storage Engine & Execution Engine Same Process • Process Crash Requires Network and/or Disk I/O to Re-read Data ©2017 Alluxio, Inc. All Rights Reserved 19 Mesos
  • 20.
    Data Resilience DuringCrash Spark Compute Spark Storage HDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Alluxio block 1 block 3 block 4 Storage Engine & Execution Engine Different process ©2017 Alluxio, Inc. All Rights Reserved 20 Mesos
  • 21.
    Data Resilience DuringCrash Process Crash - Data is Re-read at Memory SpeedHDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Alluxio block 1 block 3 block 4 CRASH Storage Engine & Execution Engine Different process ©2017 Alluxio, Inc. All Rights Reserved 21 Mesos
  • 22.
    Alluxio Architecture ©2017 Alluxio,Inc. All Rights Reserved 22 Application AlluxioClient Alluxio Master Alluxio Worker Alluxio Worker … Storage Storage …
  • 23.
    Alluxio Client ©2017 Alluxio,Inc. All Rights Reserved 23 Applications interact with Alluxio via the Alluxio client ● Native Alluxio Filesystem Client • Alluxio specific operations like [un]pin, [un]mount, [un]set TTL ● HDFS-Compatible Filesystem Client • No code change necessary ● S3 API
  • 24.
    Alluxio Master ©2017 Alluxio,Inc. All Rights Reserved 24 Master is responsible for managing metadata ● Filesystem namespace metadata ● Blocks / workers metadata Primary master writes journal for durable operations ● Secondary masters replay journal entries
  • 25.
    Alluxio Worker ©2017 Alluxio,Inc. All Rights Reserved 25 Worker is responsible for managing block data Worker stores block data on various storage media ● HDD, SSD, Memory Reads and writes data to underlying storage systems
  • 26.
    Outline Alluxio Overview Alluxio +Spark + Mesos Use Cases Using Spark with Alluxio on Mesos Deployment with Mesos Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 26
  • 27.
    Alluxio on DC/OS ©2017Alluxio, Inc. All Rights Reserved 27
  • 28.
    Alluxio on DC/OS ©2017Alluxio, Inc. All Rights Reserved 28 Alluxio brings A unified view of data across disparate storage systems High performance & predictable SLA for analytics workloads DC/OS makes provisioning infrastructure easy Automates provisioning, management & elastic scaling Benefits include: Faster analytics with Spark and other frameworks Process data from hybrid cloud storage systems (HDFS, S3, etc)
  • 29.
    Outline Alluxio Overview Alluxio +Spark + Mesos Use Cases Using Spark with Alluxio on Mesos Deployment with Mesos Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 29
  • 30.
    Demo Environment Spark Alluxio ©2017 Alluxio,Inc. All Rights Reserved 30 SPARK MESOS
  • 31.
    Demo Setup Alluxio 1.5.0 DC/OS1.9.4 Spark 2.0.2 Amazon EC2 (m3.xlarge) ©2017 Alluxio, Inc. All Rights Reserved 31
  • 32.
    Results ©2017 Alluxio, Inc.All Rights Reserved 32 8x  improvement  
  • 33.
    Conclusion Easy to useAlluxio with Spark in a Mesos environment Predictable and improved performance Easily connect to various storage systems ©2017 Alluxio, Inc. All Rights Reserved 33
  • 34.
    Thank you! Gene Pang SoftwareEngineer gene@alluxio.com 34 Twitter.com/alluxio Linkedin.com/alluxio Website www.alluxio.com E-mail info@alluxio.com @ Social Media * ©2017 Alluxio, Inc. All Rights Reserved