![0
50
100
150
200
250
0 5 10 15 20 25 30 35 40 45 50
Time[seconds]
RDD Size [GB]
Alluxio (textFile) Alluxio (objectFile) DISK_ONLY
MEMORY_ONLY_SER MEMORY_ONLY
Reading Cached RDD
©2017 Alluxio, Inc. All Rights Reserved 2 7](https://image.slidesharecdn.com/2017-08-05usingalluxiowithspark-171011211447/75/Spark-Pipelines-in-the-Cloud-with-Alluxio-by-Bin-Fan-27-2048.jpg)
![0 100 200 300 400 500 600 700 800
Alluxio
(textFile)
Alluxio
(objectFile)
No Alluxio
Time [seconds]
7x speedup
16x speedup
New Context: Read 50 GB RDD
(S3)
©2017 Alluxio, Inc. All Rights Reserved 2 8](https://image.slidesharecdn.com/2017-08-05usingalluxiowithspark-171011211447/75/Spark-Pipelines-in-the-Cloud-with-Alluxio-by-Bin-Fan-28-2048.jpg)
![Reading Cached DataFrame
(parquet)
0
50
100
150
200
250
0 5 10 15 20 25 30 35 40 45 50
Time[seconds]
DataFrame Size [GB]
Alluxio (textFile) MEMORY_ONLY_SER MEMORY_ONLY
©2017 Alluxio, Inc. All Rights Reserved 2 9](https://image.slidesharecdn.com/2017-08-05usingalluxiowithspark-171011211447/75/Spark-Pipelines-in-the-Cloud-with-Alluxio-by-Bin-Fan-29-2048.jpg)
![New Context: Read 50 GB DataFrame
(S3)
0 250 500 750 1000 1250 1500 1750
Alluxio
No Alluxio
Time [seconds]
10x average speedup, 17x peak speedup
©2017 Alluxio, Inc. All Rights Reserved 3 0](https://image.slidesharecdn.com/2017-08-05usingalluxiowithspark-171011211447/75/Spark-Pipelines-in-the-Cloud-with-Alluxio-by-Bin-Fan-30-2048.jpg)
Uploaded byData Con LA
Spark Pipelines in the Cloud with Alluxio by Bin Fan
The document presents an overview of Alluxio, a storage solution that enhances performance when used alongside Spark in big data ecosystems. It outlines various use cases, including improvements in operational efficiency and speed of data queries, showcasing case studies that demonstrate significant performance gains with Alluxio. The conclusion emphasizes Alluxio's ease of use, predictable performance, and compatibility with various storage systems.
More Related Content
Similar to Spark Pipelines in the Cloud with Alluxio by Bin Fan
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Spark Pipelines in the Cloud with Alluxio by Bin Fan
- 1. Spark Pipelines inthe Cloud with Alluxio Bin Fan, Alluxio, Inc. BIG DATA DAY LA – Aug 2017
- 2. Outline Alluxio Overview Alluxio +Spark Use Cases Using Spark with Alluxio Performance Evaluation Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 2
- 3. Data Ecosystem Yesterday •One Compute Framework • Single Storage System • Co-located ©2017 Alluxio, Inc. All Rights Reserved 3
- 4. Data Ecosystem Today •Many Compute Frameworks • Multiple Storage Systems • Most not co-located ©2017 Alluxio, Inc. All Rights Reserved 4
- 5. Data Ecosystem Issues •Each application manage multiple data sources • Add/Removing data sources require application changes • Storage optimizations requires application change • Lower performance due to lack of locality ©2017 Alluxio, Inc. All Rights Reserved 5
- 6. Data Ecosystem withAlluxio • Apps only talk to Alluxio • Simple Add/Remove • No App Changes • Highest performance in Memory • No Lock in Native File System Hadoop Compatible File System Native Key-Value Interface Fuse Compatible File System HDFS Interface Amazon S3 Interface Swift Interface GlusterFS Interface ©2017 Alluxio, Inc. All Rights Reserved 6
- 7. Next Gen Analyticswith Alluxio Native File System Hadoop Compatible File System Native Key-Value Interface Fuse Compatible File System HDFS Interface Amazon S3 Interface Swift Interface GlusterFS Interface Apps, Data & Storage at Mem Speed Big Data/IoT AI/ML Deep Learning Cloud Migration Multi Platform Autonomous ©2017 Alluxio, Inc. All Rights Reserved 7
- 8. Fastest Growing BigData Open Source Projects Fastest Growing open- source project in the big data ecosystem Running in large production clusters 500+ Contributors from 100+ organizations 0 100 200 300 400 500 0 10 20 30 40 45 NumberofContributors Github Open Source Contributors by Month Alluxio Spark Kafka Redis HDFS Cassandra Hive ©2017 Alluxio, Inc. All Rights Reserved 8
- 9. Outline Alluxio Overview Alluxio +Spark Use Cases Using Spark with Alluxio Performance Evaluation Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 9
- 10. Big Data CaseStudy – 1 010/11/2017 ©2017 Alluxio, Inc. All Rights Reserved Challenge – Gain end to end view of business with large volume of data Queries were slow / not interactive, resulting in operational inefficiency SPARK TERADATA SPARK TERADATA Solution – ETL Data from Teradata to Alluxio Impact – Faster Time to Market – “Now we don’t have to work Sundays” http://bit.ly/2oMx95W
- 11. Big Data CaseStudy – 1110/11/2017 ©2017 Alluxio, Inc. All Rights Reserved Challenge – Gain end to end view of business with large volume of data Queries were slow / not interactive, resulting in operational inefficiency SPARK Baidu File System SPARK Baidu File System Solution – With Alluxio, data queries are 30X faster Impact – Higher operational efficiency http://bit.ly/2pDHS3O
- 12. Big Data CaseStudy – Challenge – Gain end to end view of business with large volume of data for $5B Travel Site Queries were slow / not interactive, resulting in operational inefficiency SPARK HDFS Solution – With Alluxio, 300x improvement in performance Impact – Increased revenue from immediate response to user behavior Use case: http://bit.ly/2pDJdrq CEPH HDFS CEPH FLINK SPARK FLINK ©2017 Alluxio, Inc. All Rights Reserved 1 2
- 13. Machine Learning CaseStudy – 1 310/11/2017 ©2017 Alluxio, Inc. All Rights Reserved Challenge – Disparate Data both on-prem and Cloud. Heterogeneous types of data. Scaling of Exabyte size data. Slow due to disk based approach. SPARK HDFS SPARK MINIO Solution – Using Alluxio to prevent I/O bottlenecks Impact – Orders of magnitude higher performance than before. http://bit.ly/2p18ds3 MESOS
- 14. Outline Alluxio Overview Alluxio +Spark Use Cases Using Spark with Alluxio Performance Evaluation Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 1 4
- 15. Consolidating Memory Storage Engine& Execution Engine Same Process • Two copies of data in memory – double the memory used • Inter-process Sharing Slowed Down by Network / Disk I/O Spark Compute Spark Storage block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 Spark Compute Spark Storage block 1 block 3 ©2017 Alluxio, Inc. All Rights Reserved 1 5
- 16. Consolidating Memory Storage Engine& Execution Engine Different process • Half the memory used • Inter-process Sharing Happens at Memory Speed Spark Compute Spark Storage HDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Alluxio block 1 block 3 block 4 Spark Compute Spark Storage ©2017 Alluxio, Inc. All Rights Reserved 1 6
- 17. Data Resilience DuringCrash Spark Compute Spark Storage block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 Storage Engine & Execution Engine Same Process ©2017 Alluxio, Inc. All Rights Reserved 1 7
- 18. Data Resilience DuringCrash CRASH Spark Storage block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 • Process Crash Requires Network and/or Disk I/O to Re-read Data Storage Engine & Execution Engine Same Process ©2017 Alluxio, Inc. All Rights Reserved 1 8
- 19. Data Resilience DuringCrash CRASH HDFS / Amazon S3 block 1 block 3 block 2 block 4 Storage Engine & Execution Engine Same Process • Process Crash Requires Network and/or Disk I/O to Re-read Data ©2017 Alluxio, Inc. All Rights Reserved 1 9
- 20. Data Resilience DuringCrash Spark Compute Spark Storage HDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Alluxio block 1 block 3 block 4 Storage Engine & Execution Engine Different process ©2017 Alluxio, Inc. All Rights Reserved 2 0
- 21. Data Resilience DuringCrash • Process Crash – Data is Re-read at Memory Speed HDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Alluxio block 1 block 3 block 4 CRASH Storage Engine & Execution Engine Different process ©2017 Alluxio, Inc. All Rights Reserved 2 1
- 22. Accessing Alluxio DataFrom Spark Writing Data Write to an Alluxio file Reading Data Read from an Alluxio file ©2017 Alluxio, Inc. All Rights Reserved 2 2
- 23. Code Example forSpark RDDs Writing RDD to Alluxio rdd.saveAsTextFile(alluxioPath) rdd.saveAsObjectFile(alluxioPath) Reading RDD from Alluxio rdd = sc.textFile(alluxioPath) rdd = sc.objectFile(alluxioPath) ©2017 Alluxio, Inc. All Rights Reserved 2 3
- 24. Code Example forSpark DataFrames Writing to Alluxio df.write.parquet(alluxioPath) Reading from Alluxio df = sc.read.parquet(alluxioPath) ©2017 Alluxio, Inc. All Rights Reserved 2 4
- 25. Outline Alluxio Overview Alluxio +Spark Use Cases Using Spark with Alluxio Performance Evaluation Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 2 5
- 26. Experiments Spark 2.0.0 +Alluxio 1.2.0 Single worker: Amazon r3.2xlarge Comparisons: Alluxio Spark Storage Level: MEMORY_ONLY Spark Storage Level: MEMORY_ONLY_SER Spark Storage Level: DISK_ONLY ©2017 Alluxio, Inc. All Rights Reserved 2 6
- 27. 0 50 100 150 200 250 0 5 1015 20 25 30 35 40 45 50 Time[seconds] RDD Size [GB] Alluxio (textFile) Alluxio (objectFile) DISK_ONLY MEMORY_ONLY_SER MEMORY_ONLY Reading Cached RDD ©2017 Alluxio, Inc. All Rights Reserved 2 7
- 28. 0 100 200300 400 500 600 700 800 Alluxio (textFile) Alluxio (objectFile) No Alluxio Time [seconds] 7x speedup 16x speedup New Context: Read 50 GB RDD (S3) ©2017 Alluxio, Inc. All Rights Reserved 2 8
- 29. Reading Cached DataFrame (parquet) 0 50 100 150 200 250 05 10 15 20 25 30 35 40 45 50 Time[seconds] DataFrame Size [GB] Alluxio (textFile) MEMORY_ONLY_SER MEMORY_ONLY ©2017 Alluxio, Inc. All Rights Reserved 2 9
- 30. New Context: Read50 GB DataFrame (S3) 0 250 500 750 1000 1250 1500 1750 Alluxio No Alluxio Time [seconds] 10x average speedup, 17x peak speedup ©2017 Alluxio, Inc. All Rights Reserved 3 0
- 31. Outline Alluxio Overview Alluxio +Spark Use Cases Using Spark with Alluxio Performance Evaluation Demo 1 2 3 4 5 ©2017 Alluxio, Inc. All Rights Reserved 3 1
- 32. Demo Environment Spark Alluxio ©2017 Alluxio,Inc. All Rights Reserved 3 2
- 33. Conclusion Easy to useAlluxio with Spark Predictable and improved performance Easily connect to various storages ©2017 Alluxio, Inc. All Rights Reserved 3 3
- 34. Thank you! Gene PangCheng Chang gene@alluxio.com cc@alluxio.com Twitter: @unityxx Twitter: @uronce Twitter.com/alluxio Linkedin.com/alluxio Website www.alluxio.com E-mail info@alluxio.com @ Social Media ©2017 Alluxio, Inc. All Rights Reserved 3 4
Editor's Notes
- #11 https://pixabay.com/en/fruits-fruit-tropical-fruits-655679/
- #12 https://pixabay.com/en/fruits-fruit-tropical-fruits-655679/
- #13 https://pixabay.com/en/fruits-fruit-tropical-fruits-655679/
- #14 https://pixabay.com/en/fruits-fruit-tropical-fruits-655679/
- #16 First I want to explain how Alluxio consolidate memory. Normally with Spark, different jobs have separate storage blocks. This causes data duplication which reduces the effective amount of memory. In addition, fetching the data is usually expensive and that is also doubled.
- #17 By using Alluxio, we reduce the number of data fetches from the slow storage and consolidate the memory used. As an additional benefit, Spark executors can be run with much less memory than before, reducing the chances of GC issues or OOM errors.
- #18 Even if there is only one Spark job, Alluxio still provides benefits in the case of crashes. Although crashes are never intended, in the big data world, we design architectures for them since they are inevitable.
- #20 When Spark crashes, both the storage and compute are gone.
- #21 With Alluxio in the picture, Spark crashes no longer cause data to be lost from memory.
- #23 Now let’s talk about how to write our Spark applications to leverage Alluxio. It is fairly straightforward. The interface is a file system interface, so reading and writing data is simply reading and writing Alluxio files.
- #24 For RDDs, it translates to these APIs. It is recommended to use text files when possible as that will get you the best performance. I will talk more about these two APIs in the performance evaluation portion.
- #25 Spark also provides APIs for more structured data in the form of dataframes. It is equally simple to read and write them with Alluxio, simply replace the original path with an Alluxio path.
- #27 We ran this experiment when Spark 2.0 came out with the latest version of Alluxio at the time. Since then we have kept track of improvements to Spark and Alluxio, but have not seen enough of a difference to rerun this performance evaluation. We compare several storage types in Spark with not storing the data in Spark but in Alluxio instead.
- #28 Let me explain what this graph is presenting. We first tested with a cached RDD, meaning the data was already in Spark or Alluxio, and we varied the size of the RDD and recorded how much time it took to run a scan on the RDD. The disk only line in green is as expected, much worse than all the others which are using memory. A more interesting observation is the performance difference between the Alluxio text file and object file. Text file is almost strictly better, this is because of the object serialization overhead, the file itself is all plaintext so using an object file was not helpful. I would recommend to use text file when possible. Using spark’s memory only is the best for small files, but abruptly performs worse once the data cannot be completely cached in memory. This happens as well for mem_only_ser the purlple line, but much last because of the small size of serialized objects. Alluxio scales linearly throughout the test and actually outperforms Spark caches for a single task after 32 GB or so.
- #29 The previous comparison was if the data was on SSD, which is still a relatively fast storage. If we instead put the data in S3, we see a much larger speed up. This is more representative of architectures because this allows compute and storage to be decoupled. In this case, we see 16x speed up even in this simple job, which is similar to some of the performance use cases I previously presented.
- #30 We also did a similar test with a parquet file using the data frame API. Here we did a simple aggregation which would access all the rows. The behavior is the same as before, using Spark’s native caching has an abrupt turning point where the performance degrades. These tests are run with default spark configurations which they suggest not to change. If we optimize for additional storage, we can move the point of bend, but it would still be present.
- #31 This is average of 7 runs. Range of s3 is 1132.765125 Range of Alluxio is 10.5890684 We also ran the same example against S3, and the variation in the test was fairly large. This is similar to what some users see in their storage either due to the storage itself, or their workload and sometimes both. Alluxio performance is much more consistent and provided on average 10x and up to 17x performance improvement.
- #33 With Alluxio in the picture, Spark crashes no longer cause data to be lost from memory.
- #34 Easy to use with spark