The Missing Piece of On-Demand Clusters Presented by Calvin Jia, Alluxio Introduction to Alluxio Meetup at Princeton http://www.meetup.com/futureofdata-princeton/events/232927731/

1. The Missing Piece of On-Demand Clusters
October 2016
Calvin Jia

2. About Me
• Calvin Jia
• Software Engineer @ Alluxio, Inc.
• Alluxio PMC
• #1 Alluxio Contributor
• Twitter: @JiaCalvin
2

3. Outline
• Cloud Architectures
• Alluxio in Cloud Architectures
• Takeaways
3

4. Cloud Architectures – Overview
• Mostly service based, from providers
– Amazon Web Services
– Google Cloud Platform
• Separate compute and storage clusters
• Compute clusters are ephemeral
4

5. Cloud Architectures – Pros & Cons
Pros
• Low maintenance
• Pay as you go
• Elastic and scalable
• Cost effective storage
Cons
• Lower Performance
5

6. Outline
• Cloud Architectures
• Alluxio in Cloud Architectures
• Takeaways
6

7. Alluxio in Cloud Architectures – Overview
• Deployed in compute clusters
– Memory speed data access
– Transparent data access to any storage
• Simple to deploy
– Mount storage systems similar to local disks
7

8. Alluxio in Cloud Architectures – Example
8

9. Alluxio in Cloud Architectures – Benefits
9
• Remedies the performance drawback
• Acceleration due to memory-speed I/O
• Designed to improve the affinity of compute
and storage

10. Alluxio in Cloud Architectures – Data Path
10
FAST
104 - 105 MB/s
MODERATE 103 - 104 MB/s
SLOW 102 - 103 MB/s
Only when necessary
Limited
Often
SSD
HDD
Mem

11. Takeaways – Experiment Results
11
0
100
200
300
400
500
600
Initial
Read Subsequent
Read Read
from
Separate
Job
Runtime
(seconds)
Spark
-­‐ No
Persist
Spark
-­‐ Persist
Alluxio

12. Takeaways – Alluxio & Cloud Architectures
12
• Cloud architectures have significant upsides
• Alluxio alleviates the major downsides

13. Try out Alluxio 1.2.0
http://www.alluxio.org/releases
13

14. Read the blog athttp://www.alluxio.com/2016/08/accelerating-on-demand-data-analytics-with-alluxio
Read the whitepaper at www.alluxio.com.
Check out www.alluxio.org.
Thank you!
14