Download to read offline
![An Example
1. lines = spark.textFile(“hdfs://…”)
2. errors = lines.filter( _.startsWith(“ERROR”) )
3. errors.persist()
4. errors.filter( _.contains(“HDFS”) ).map( _.split(‘t’)[3]).count
26](https://image.slidesharecdn.com/sparkpresentation-151219094621/85/Apache-Spark-26-320.jpg)
![An Example
27
lines
Errors
HDFS
errors
Part3
filter(_.startsWith(“ERROR”))
filter(_.contains(“HDFS”))
map( _.split(‘t’)[3])](https://image.slidesharecdn.com/sparkpresentation-151219094621/85/Apache-Spark-27-320.jpg)
Uploaded byMahdi Esmailoghli
Apache Spark
This document discusses Resilient Distributed Datasets (RDD), a fault-tolerant abstraction in Apache Spark for cluster computing. RDDs allow data to be reused across computations and support transformations like map, filter, and join. RDDs can be created from stable storage or other RDDs, and Spark computes them lazily for efficiency. The document provides examples of how RDDs can express algorithms like MapReduce, SQL queries, and graph processing. Benchmarks show Spark is 20x faster than Hadoop for iterative algorithms due to RDDs enabling data reuse in memory across jobs.
More Related Content
Apache Spark
- 1. Resilient Distributed Dataset:A Fault-Tolerant Abstraction For In-Memory Cluster Computing Mahdi Esmail oghli Dr. Bagheri AmirKabir University of technology SPARK http://BigData.ceit.aut.ac.ir
- 2.
- 3.
- 4. Dealing With BigData 4 Sampling Hashing Approximationmethods Map-Reduce Model …
- 5. Map-Reduce Model 5 A BigDataProgramming Model Potential of parallelism Can be executed on a cluster
- 6.
- 7. Problems with currentcomputing frameworks Specially Map-reduce Provides abstraction for accessing the cluster’s computational resources Lack of abstraction for distributed memory 7
- 8. Problems with currentcomputing frameworks Specially Map-reduce Makes them inefficient for those that reuse intermediate results across multiple computations 8
- 9. SPARK Motivation Problems withcurrent computing frameworks (ex. Map-Reduce): Iterative algorithms Interactive data mining tools 9
- 10. Data reuse examples Iterativemachine learning and graph algorithms: Page Rank K-means clustering Logistic regression 10
- 11. Data reuse examples InteractiveData Mining (runs multiple queries on the same subset of data) : Statistical queries Fraud detection Stream queries 11
- 12. Current Solution The onlyway to reuse data between computations with current frameworks: Write it to an external stable storage system. X 12
- 13.
- 14. Developed System For ReusingIntermediate Data Pregel : Iterative graph computation HALOOP: Iterative map-reduce interface 14
- 15. Developed System For ReusingIntermediate Data Just for specific computation patterns. We need abstraction for more general reuse. 15
- 16.
- 17. RDD Read-Only partitioned collectionof records. Can be created on either stable storage or other RDDs (using transformations). User can control Persistence and Partitioning 17
- 18. RDD Efficient data reuse Paralleldata structure Allow explicit persist results In-memory computation Large clusters fault-tolerant manner 18
- 19. Current fault tolerantapproaches Data replication across machines Log update across machines 19
- 20. 2 Main interfacefor RDD 20 RDD Actions Transformations
- 21. -e.g., map, filterand join Transformations 21 Interface used for fault tolerance in RDD
- 22. Actions SPARK computes RDDsLazily (Helps pipelining) Actions return value. ex. Count - Collect - Save 22
- 23. RDD can expressother models Map - Reduce SQL Pregel HALOOP … 23
- 24.
- 25.
- 26. An Example 1. lines= spark.textFile(“hdfs://…”) 2. errors = lines.filter( _.startsWith(“ERROR”) ) 3. errors.persist() 4. errors.filter( _.contains(“HDFS”) ).map( _.split(‘t’)[3]).count 26
- 27.
- 28. Persistent Function Indicates whichRDDs we want to reuse in the future actions. Other persistence strategies like: Store the RDD only in disk Replicating across machines Set persistence priorities to RDDs. 28
- 29. SPARK RDD is been implementedin a system called SPARK In SCALA Language 29
- 30. What benchmarks show aboutSPARK 20X faster than HADOOP for iterative applications It can scan 1TB dataset with 5-7s latency 30 100 GB Data 100 node
- 31. Evaluation(Logistic Regression) 31 0 35 70 105 140 HADOOP HADOOPBMSPARK 3 62 76 46 139 80 First Iteration Later Iterations
- 32. Evaluation(K-means) 32 0 50 100 150 200 HADOOP HADOOPBM SPARK 33 87 106 82 182 115 FirstIteration Later Iterations
- 33. SPARK STACK 33 Apache Spark DistributedFile System. e.x. HDFS, GlusterFS Spark SQL Spark Streaming MLlib GraphX
- 34. SPARK Won DaytonaGraySort Contest 2014 Spark officially sets a new record in large-scale sorting 34
- 35. Spark the fastestopen source engine for sorting 35 HADOOP MR SPARK SPARK 1PT Data Size 102.5 TB 100 TB 1000 TB Elapsed Time 72 mins 23 mins 234 mins # Nodes 2100 206 190 # Cores 50400 physical 6592 virtualized 6080 virtualized Cluster disk throughput 3150 GB/s 618 GB/s 570 GB/s Environment Dedicated datacenter EC2 (i2.8xlarge) EC2 (i2.8xlarge) Sort rate 1.42 PT/min 4.27 TB/min 4.27 TB/min Without using Spark’s in-memory cache
- 36.
- 37. SPARK > HADOOPMR 37
- 38. References 38 Dean, Jeffrey, andSanjay Ghemawat. “MapReduce: simplified data processing on large clusters." Communications of the ACM 51.1 (2008): 107-113. Zaharia, Matei, et al. "Resilient distributed datasets: A fault- tolerant abstraction for in-memory cluster computing." Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation. USENIX Association, 2012. http://Spark.apache.org https://databricks.com
- 39.