The document provides an overview of Apache Spark, emphasizing its ability to run programs significantly faster than MapReduce, with speeds up to 100x in memory and 10x on disk. It highlights Spark's functionalities, including resilient distributed datasets (RDDs), Spark Streaming for real-time data processing, and related technologies such as Google Cloud Dataflow. Additionally, the document touches on performance optimization techniques and mentions various resources and APIs associated with Spark.

1. SPARK
Alexey Diomin, diominay@gmail.com

2. Intro

3. Basic
 RDD
 DAG

4. RDD
 Resilient Distributed Dataset

5. RDD
 Resilient Distributed Dataset
 SchemaRDD

6. DAG

7. DAG

8. DAG

9. Mythology
 Spark is not MapReduce

10. Mythology
 Spark is not MapReduce
 Run programs up to 100x faster than
MapReduce in memory, or 10x faster on disk

11. Mythology
 Spark is not MapReduce
 Run programs up to 100x faster than
MapReduce in memory, or 10x faster on disk
 InMemory processing

12. Mythology
 Spark is not MapReduce
 Run programs up to 100x faster than
MapReduce in memory, or 10x faster on disk
 InMemory processing
 Spark Streaming is real-time streaming

13. Mythology
 Spark is not MapReduce
 Run programs up to 100x faster than
MapReduce in memory, or 10x faster on disk
 InMemory processing
 Spark Streaming is real-time streaming
 Lightning-fast cluster computing

14. MapReduce

15. MapReduce

16. MapReduce

17. Not MapReduce

18. Spark
 Run programs up to 100x faster than
MapReduce in memory, or 10x faster on disk

19. Spark
 Run programs up to 100x faster than Hadoop
MapReduce* in memory, or 10x faster on disk
*Hadoop without Tez

20. http://spark.apache.org/

21. InMemory

22. InMemory
 The MapReduce and Spark shuffles use a “pull”
model. Every map task writes out data to local
disk, and then the reduce tasks make remote
requests to fetch that data
 http://blog.cloudera.com/blog/2015/01/improving-sort-performance-in-apache-spark-its-a-double/

23. Spark Streaming
 RDD
 DAG

24. Spark Streaming

25. Spark Streaming
Receiver.store(...)

26. Spark Streaming

27. Google Cloud Dataflow
 One of the most compelling aspects of Cloud
Dataflow is its approach to one of the most
difficult problems facing data engineers: how to
develop pipeline logic that can execute in both
batch and streaming contexts.
 http://blog.cloudera.com/blog/2015/01/new-in-cloudera-labs-google-cloud-
dataflow-on-apache-spark/

28. Lightning-fast cluster computing

29. Lightning-fast cluster computing

30. Lightning-fast cluster computing

31. Lightning-fast cluster computing

32. Lightning-fast cluster computing

33. Spark
 Logging
 Pipeline
 Indexes
 Job progress
 Effective Memory
 Network

34. Example

35. Staged (batch) execution

36. Pipelined execution

37. Indexes
 Netflix
 https://github.com/amplab/spark-indexedrdd

38. Job Progress
 Accumulators
 Broadcast

39. Memory
 val value = task.run(taskId, attemptNumber)

40. Memory
 val value = task.run(taskId, attemptNumber)
 val valueBytes = resultSer.serialize(value)

41. Memory
 val value = task.run(taskId, attemptNumber)
 val valueBytes = resultSer.serialize(value)
 val directResult = new DirectTaskResult(valueBytes,
accumUpdates, task.metrics.orNull)
 val serializedDirectResult = ser.serialize(directResult)

42. Memory
 val value = task.run(taskId, attemptNumber)
 val valueBytes = resultSer.serialize(value)
 val directResult = new DirectTaskResult(valueBytes,
accumUpdates, task.metrics.orNull)
 val serializedDirectResult = ser.serialize(directResult)
 Default JavaSerializer
public synchronized byte toByteArray()[] {
return Arrays.copyOf(buf, count);
}

43. Network

44. Network

45. Network
 Problem with firewall/nat/multiple ip/etc.

46. SQL
 Shark (dead)
 Spark SQL
 Spark on Hive

47. SparkR

48. SparkR
 Unstable API
 Minimum docs

49. SparkR
 Unstable API
 Minimum docs
 Rstudio Server

50. Links
 Spark
 http://spark.apache.org/
 Flink
 http://flink.apache.org/
 Tez
 http://tez.apache.org/