Best Practices for Using Alluxio with Spark

Best Practices for Using
Alluxio with Spark
Gene Pang,Alluxio, Inc.
Spark Summit EU - October 2017

About Me
• Gene Pang
• Software engineer @ Alluxio, Inc.
• Alluxio open source PMC member
• Ph.D. from AMPLab @ UC Berkeley
• Worked at Google before UC Berkeley
• Twitter: @unityxx
• Github: @gpang
©2017 Alluxio, Inc.All Rights Reserved 2

Outline
Alluxio Overview
Alluxio + Spark Use Cases
Alluxio Architecture
Using Spark with Alluxio
Experiments
1
2
3
4
5

Data EcosystemYesterday
• One Compute
Framework
• Single Storage System
• Co-located

Data Ecosystem Today
• Many Compute
Frameworks
• Multiple Storage Systems
• Most not co-located

Data Ecosystem Issues
• Each application manage
multiple data sources
• Add/Removing data
sources require
application changes
• Storage optimizations
requires application
change
• Lower performance due
to lack of locality

Data Ecosystem with Alluxio
• Apps only talk to
Alluxio
• Simple Add/Remove
• No App Changes
• Memory
Performance
Native File System
Hadoop Compatible
File System
Native Key-Value
Interface
Fuse Compatible File
System
HDFS Interface Amazon S3 Interface Swift Interface GlusterFS Interface

Next Gen Analytics with Alluxio
Native File System
Hadoop Compatible
File System
Native Key-Value
Interface
Fuse Compatible File
System
HDFS Interface Amazon S3 Interface Swift Interface GlusterFS Interface
Apps, Data & Storage
at Memory Speed
ü Big Data/IoT
ü AI/ML
ü Deep Learning
ü Cloud Migration
ü Multi Platform
ü Autonomous

Fastest Growing Big Data
Open Source Projects
Fastest Growing open-
source project in the big
data ecosystem
Running in large
production clusters
600+ Contributors from
100+ organizations
0
100
200
300
400
500
0 10 20 30 40 45
NumberofContributors
Github Open Source Contributors by Month
Alluxio
Spark
Kafka
Redis
HDFS
Cassandra
Hive

Outline
Alluxio Overview
Experiments
1
2
3
4
5
©2017 Alluxio, Inc.All Rights Reserved 1 0

Big Data Case Study –
1 19/17/19 ©2017 Alluxio, Inc.All Rights Reserved
Challenge –
Gain end to end view of
business with large volume of
data
Queries were slow / not
interactive, resulting in
operational inefficiency
SPARK
TERADATA
SPARK
TERADATA
Solution –
ETL Data from Teradata to Alluxio
Impact –
Faster Time to Market – “Now we
don’t have to work Sundays”
http://bit.ly/2oMx95W

Challenge –
data
SPARK
Baidu File System
SPARK
Baidu File System
Solution –
With Alluxio, data queries are 30X
faster
Impact –
Higher operational efficiency
http://bit.ly/2pDHS3O

Challenge –
data for $5B Travel Site
SPARK
HDFS
Solution –
With Alluxio, 300x improvement in
performance
Impact –
Increased revenue from immediate
response to user behavior
Use case: http://bit.ly/2pDJdrq
CEPH
HDFS CEPH
FLINK SPARK FLINK

Machine Learning Case Study –
Challenge –
Disparate Data both on-prem
and Cloud. Heterogeneous
types of data.
Scaling of Exabyte size data.
Slow due to disk based
approach.
SPARK
HDFS
SPARK
MINIO
Solution –
Using Alluxio to prevent I/O
bottlenecks
Impact –
Orders of magnitude higher
performance than before.
http://bit.ly/2p18ds3
MESOS

Outline
Alluxio Overview
Experiments
1
2
3
4
5

1 6©2017 Alluxio, Inc.All Rights Reserved
Application
AlluxioClient
Alluxio
Master
Alluxio
Worker
Alluxio
Worker
…
Storage
Storage
…

Alluxio Client
Applications interact with Alluxio via the Alluxio client
• Java Native Alluxio Filesystem Client
• Alluxio specific operations like [un]pin, [un]mount, [un]set TTL
• HDFS-Compatible Filesystem Client
• No code change necessary
• S3 API

Alluxio Master
Master is responsible for managing metadata
• Filesystem namespace metadata
• Blocks / workers metadata
Primary master writes journal for durable operations
• Secondary masters replay journal entries

Alluxio Worker
Worker is responsible for managing block data
Worker stores block data on various storage media
• HDD, SSD, Memory
Reads and writes data to underlying storage systems

Outline
Alluxio Overview
Experiments
1
2
3
4
5

Sharing Data via Memory
Storage Engine &
Execution Engine
Same Process
• Two copies of data in memory – double the memory used
• Sharing Slowed Down by Network / Disk I/O
Spark Compute
Spark
Storage
block 1
block 3
HDFS / Amazon S3
block 1
block 3
block 2
block 4
Spark Compute
Spark
Storage
block 1
block 3

Sharing Data via Memory
Storage Engine &
Execution Engine
Different process
• Half the memory used
• Sharing Data at Memory Speed
Spark Compute
Spark Storage
HDFS / Amazon S3
block 1
block 3
block 2
block 4
HDFS
disk
block 1
block 3
block 2
block 4
Alluxio
block 1
block 3 block 4
Spark Compute
Spark Storage

Data Resilience During Crash
Spark Compute
Spark Storage
block 1
block 3
HDFS / Amazon S3
block 1
block 3
block 2
block 4
Storage Engine &
Execution Engine
Same Process

CRASH
Spark Storage
block 1
block 3
HDFS / Amazon S3
block 1
block 3
block 2
block 4
• Process Crash Requires Network and/or Disk I/O to Re-read Data
Storage Engine &
Execution Engine
Same Process

CRASH
HDFS / Amazon S3
block 1
block 3
block 2
block 4
Storage Engine &
Execution Engine
Same Process
• Process Crash Requires Network and/or Disk I/O to Re-read Data

Spark Compute
Spark Storage
HDFS / Amazon S3
block 1
block 3
block 2
block 4
HDFS
disk
block 1
block 3
block 2
block 4
Alluxio
block 1
block 3 block 4
Storage Engine &
Execution Engine
Different process

• Process Crash – Data is Re-read at Memory Speed
HDFS / Amazon S3
block 1
block 3
block 2
block 4
HDFS
disk
block 1
block 3
block 2
block 4
Alluxio
block 1
block 3 block 4
CRASH Storage Engine &
Execution Engine
Different process

Accessing Alluxio Data From Spark
Writing Data Write to an Alluxio file
Reading Data Read from an Alluxio file

Code Example for Spark RDDs
Writing RDD to Alluxio
rdd.saveAsTextFile(alluxioPath)
rdd.saveAsObjectFile(alluxioPath)
Reading RDD from Alluxio
rdd = sc.textFile(alluxioPath)
rdd = sc.objectFile(alluxioPath)

Code Example for Spark DataFrames
Writing to Alluxio df.write.parquet(alluxioPath)
Reading from Alluxio df = sc.read.parquet(alluxioPath)

Deploying Alluxio with Spark
Spark
Alluxio
Storage
Spark
Alluxio
Storage
Colocate Alluxio Workers with Spark
for optimal I/O performance
Deploy Alluxio between
Spark and Storage

Outline
Alluxio Overview
Experiments
1
2
3
4
5

Experiments
Spark 2.2.0 + Alluxio 1.6.0
Single worker:Amazon r3.2xlarge
Compare reading cached parquet files

Reading Cached DataFrame (parquet)

New Context: 50 GB DataFrame (S3)
6x – 8x speedup

Conclusion
Easy to use Alluxio with Spark
Alluxio enables improved I/O performance
Easily interact with various storage systems with Alluxio

Thank you!
Gene Pang
gene@alluxio.com
Twitter: @unityxx
Twitter.com/alluxio
Linkedin.com/alluxio
Website
www.alluxio.com
E-mail
info@alluxio.com
@
Social Media
á
™

Best Practices for Using Alluxio with Spark

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Best Practices for Using Alluxio with Spark

Similar to Best Practices for Using Alluxio with Spark (20)

More from Alluxio, Inc.

More from Alluxio, Inc. (20)

Recently uploaded

Recently uploaded (20)

Best Practices for Using Alluxio with Spark