The document introduces Resilient Distributed Datasets (RDDs), a fault-tolerant in-memory computing abstraction designed for efficient data processing in cluster environments, particularly for tasks like machine learning and real-time data mining. RDDs enable efficient data reuse, lazy computation, and user control over data persistence while addressing high I/O latency and fault recovery issues. Implemented in Apache Spark, RDDs leverage a lineage tracking system to optimize performance through task scheduling and recovery from failures.

1. RESILIENT DISTRIBUTED DATASETS: A FAULT-TOLERANT ABSTRACTION FOR IN-MEMORY CLUSTER COMPUTING
MateiZahariaet al.
Universityof California, Berkeley

2. Alessandro MenabòPolitecnico di Torino, Italy

3. INTRODUCTION

4. Motivations
Interactive (real-time) data mining
Reuseof intermediate results(iterative algorithms)
Examples:
Machine learning
K-meansclustering
PageRank

5. Limitationsof currentframeworks
Data reuseusuallythroughdisk storage
Disk IO latencyand serialization
Too high-levelabstractions
Implicitmemorymanagement
Implicitwork distribution
Fault tolerancethroughdata replicationand logging
High network traffic

6. Goals
Keepfrequentlyuseddata in mainmemory
Efficientfault recovery
Log data transformationsratherthandata itself
User control

7. RESILIENT DISTRIBUTED DATASETS (RDDs)

8. Whatisan RDD?
Read-only, partitionedcollectionof recordsin key-valueform
Createdthroughtransformations
From storeddata or otherRDDs
Coarse-grained: sameoperationon the wholedataset
Examples: map, filter, join
Lineage: sequenceof transformationsthatcreatedthe RDD
Keyto efficientfault recovery
Usedthroughactions
Return a resultor storedata
Examples: count, collect, save

9. Whatisan RDD? (cont’d)
Lazycomputation
RDDsare computedonlywhenthe first actionisinvoked
Persistencecontrol
ChooseRDDsto be reused, and howto storethem(e.g. in memory)
Partitioningcontrol
Definehowto distributeRDDsacrosscluster nodes
Minimizeinter-nodecommunication

10. Implementation
Apache Sparkcluster computingframework
Open source
Basedon HadoopDistributed File System (HDFS) (by Apache)
Scala programminglanguage
Derivedfrom Java, compilesto Java bytecode
Object-orientedand functionalprogramming
Staticallytyped, efficientand concise

11. Sparkprogramminginterface
Driver program
Definesand invokesactionson RDDs
TracksRDDs’ lineage
Assignsworkloadto workers
Workers
Persistentprocesseson cluster nodes
Performactionson data
Can storepartitionsof RDDsin RAM

12. Example: PageRank
Iterative algorithm
Updatesdocumentrankbasedon contributionsfrom documentsthatlink to it

13. Example: PageRank(cont’d)
The graphgrowswith the numberof iterations
Replicate some intermediate resultsto speedupfault recovery
Reduce communicationoverhead
Partitionbothlinksand ranksby URL in the sameway
Joiningthemcan be doneon the samenode

14. RDD representation
Goals
Easilytracklineage
Supportrichset of transformations
Keepsystemassimpleaspossible(uniforminterface, avoidad-hoc logic)
Graph-basedstructure
Set of partitions(piecesof the dataset)
Set of dependencieson parentRDDs
Functionfor computingthe datasetfrom parentRDDs
Metadataaboutpartitioningand data location

15. Dependencies
Narrowdependencies
Eachpartitionof the parentisusedby atmostonepartitionof the child
Example: map, filter, union
Wide dependencies
Eachpartitionof the parentmaybe usedby manypartitionsof the child
Example: join, groupByKey

16. Dependencies(cont’d)
Normalexecution
Narrowpipelined(e.g. map+ filteroneelementata time)
Wide serial (allparentsneedto be availablebeforecomputationstarts)
Fault recovery
Narrowfast (onlyoneparentpartitionhasto be recomputed)
Wide full (onefailednodemayrequireallparentsto be recomputed)

17. OVERVIEW OF SPARK

18. Scheduling
Tracksin-memorypartitions
On actionrequest:
Examineslineageand buildsa DAG of executionstages
Eachstage containsasmanytransformationswith narrowdependenciesaspossible
Stage boundariescorrespondto wide dependencies, or alreadycomputedpartitions
Launchestasksto compute missingpartitionsuntildesiredRDD iscomputed
Tasksassignedaccordingto in-memorydata locality
Otherwiseassignto RDD’spreferredlocation (user-specified)

19. Scheduling(cont’d)
On task failure, re-runiton anothernodeifallparentsare stillavailable
Ifstagesbecomeunavailable, re-runparenttasksin parallel
Schedulerfailuresnotaddressed
Replicate lineagegraph?

20. Interactivity
Desirablegivenlow-latencyin-memorycapabilities
Scala shellintegration
Eachline iscompiledintoa Java classand runin JVM
Bytecodeshippedto workersvia HTTP

21. Memory management
PersistentRDDsstoragemodes:
In-memory, deserializedobject: fastest(native supportby JVM)
In-memory, serializedobject: more memory-efficient, butslower
On-disk: ifRDD doesnotfitintoRAM, buttoocostlyto recomputeeverytime
LRU evictionpolicy of entireRDD whennew partitiondoesnotfitintoRAM
Unlessthe new partitionbelongsto the LRU RDD
Separate memoryspaceon eachnode

22. Checkpointing
Save intermediate RDDsto disk (replication)
Speeduprecoveryof RDDswith long lineageor wide dependencies
Pointlesswith short lineageor narrowdependencies(recomputingpartitionsin parallelislesscostlythanreplicatingthe wholeRDD)
Notstrictlyrequired, butniceto have
Easy becauseRDDsare read-only
No consistencyissuesor distributedcoordinationrequired
Donein the background, programsdo nothaveto be suspended
Controlledby the user, no automaticcheckpointingyet

23. EVALUATION

24. Testingenvironment
Amazon ElasticCompute Cloud(EC2)
m1.xlarge nodes
4 cores/ node
15 GB of RAM / node
HDFS with 256 MB blocks

25. Iterative machine learning
10 iterationson 100 GB of data
Runon 25, 50, 100 nodes

26. Iterative machine learning(cont’d)
Differentalgorithms
K-meansismore compute-intensive
Logisticregressionismore sensitive to IO and deserialization
Minimum overheadin Spark
25.3×/ 20.7×with logisticregression
3.2×/ 1.9×with K-means
OutperformsevenHadoopBinMem(in-memorybinarydata)

27. PageRank
10 iterationson a 54 GB Wikipedia dump
Approximately4 millionarticles
Runon 30 and 60 nodes
Linear speedupwith numberof nodes
2.4×with in-memorystorageonly
7.4×with partitioncontrollingtoo

28. Fault recovery
10 iterationsof K-meanswith 100 GB of data on 75 nodes
Failureat6thiteration

29. Fault recovery(cont’d)
Lossof tasksand partitionson failednode
Task rescheduledon differentnodes
Missingpartitionsrecomputedin parallel
Lineagegraphslessthan10 KB
Checkpointingwouldrequire
Runningseveraliterationsagain
Replicate all100 GB over the network
Consumetwicethe memoryor writeall100 GB to disk

30. Lowmemory
Logisticregressionwith variousamountsof RAM
Gracefuldegradationwith lessspace

31. Interactive data mining
1 TB of Wikipedia page viewlogs(2 yearsof data)
Runon 100 m2.4xlarge nodes
8 coresand 68 GB of RAM per node
True interactivity(lessthan7 s)
Queryingfrom disk took170 s

32. CONCLUSIONS

33. Applications
Nothingnew under the sun
In-memorycomputing, lineagetracking, partitioningand fast recoveryare alreadyavailablein otherframeworks(separately)
RDDscan provideallthesefeaturesin a single framework
RDDscan express existingcluster programmingmodels
Sameoutput, betterperformance
Examples: MapReduce, SQL, Google’sPregel, batchedstreamprocessing (periodicallyupdatingresultswith new data)

34. Advantages
Dramaticspeedupwith reuseddata (dependingon application)
Fast fault recoverythanksto lightweightloggingof transformations
Efficiencyunder control of user(storage, partitioning)
Gracefulperformance degradationwith lowRAM
High expressivity
Versatility
Interactivity
Open source 

35. Limitations
Notsuitedfor fine-grainedtransformations
Overheadfrom loggingtoomanylineagegraphs
Traditionaldata loggingand checkpointingperformbetter

36. Thanks!