Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Missing Piece of On-Demand Clusters

765 views

Published on

The Missing Piece of On-Demand Clusters

Presented by Calvin Jia, Alluxio
Introduction to Alluxio Meetup at Princeton
http://www.meetup.com/futureofdata-princeton/events/232927731/

Published in: Software
  • Be the first to comment

  • Be the first to like this

The Missing Piece of On-Demand Clusters

  1. 1. The Missing Piece of On-Demand Clusters October 2016 Calvin Jia
  2. 2. About Me • Calvin Jia • Software Engineer @ Alluxio, Inc. • Alluxio PMC • #1 Alluxio Contributor • Twitter: @JiaCalvin 2
  3. 3. Outline • Cloud Architectures • Alluxio in Cloud Architectures • Takeaways 3
  4. 4. Cloud Architectures – Overview • Mostly service based, from providers – Amazon Web Services – Google Cloud Platform • Separate compute and storage clusters • Compute clusters are ephemeral 4
  5. 5. Cloud Architectures – Pros & Cons Pros • Low maintenance • Pay as you go • Elastic and scalable • Cost effective storage Cons • Lower Performance 5
  6. 6. Outline • Cloud Architectures • Alluxio in Cloud Architectures • Takeaways 6
  7. 7. Alluxio in Cloud Architectures – Overview • Deployed in compute clusters – Memory speed data access – Transparent data access to any storage • Simple to deploy – Mount storage systems similar to local disks 7
  8. 8. Alluxio in Cloud Architectures – Example 8
  9. 9. Alluxio in Cloud Architectures – Benefits 9 • Remedies the performance drawback • Acceleration due to memory-speed I/O • Designed to improve the affinity of compute and storage
  10. 10. Alluxio in Cloud Architectures – Data Path 10 FAST 104 - 105 MB/s MODERATE 103 - 104 MB/s SLOW 102 - 103 MB/s Only when necessary Limited Often SSD HDD Mem
  11. 11. Takeaways – Experiment Results 11 0 100 200 300 400 500 600 Initial  Read Subsequent  Read Read  from   Separate  Job Runtime  (seconds) Spark  -­‐ No  Persist Spark  -­‐ Persist Alluxio
  12. 12. Takeaways – Alluxio & Cloud Architectures 12 • Cloud architectures have significant upsides • Alluxio alleviates the major downsides
  13. 13. Try out Alluxio 1.2.0 http://www.alluxio.org/releases 13
  14. 14. Read the blog athttp://www.alluxio.com/2016/08/accelerating-on-demand-data-analytics-with-alluxio Read the whitepaper at www.alluxio.com. Check out www.alluxio.org. Thank you! 14

×