This document discusses Apache Spark, Spark Streaming, and Tachyon for solving big data problems. It defines Spark as a fast engine for large-scale data processing, Spark Streaming for building scalable streaming applications, and Tachyon as a memory-centric distributed file system. The document also covers Spark and Tachyon architectures, distributed execution in Spark, limitations of MapReduce, fault tolerance in Spark Streaming, and data locality in Spark.

1. Spark, Spark Streaming
& Tachyon
Solving big data problem without programming for big data

2. Who am I? What do we do?
• Name: Johan Hong johan.hong@pearson.com
• Software Architect work for Pearson Higher Education
• Deliver personalized and connected learning at scale
• Build assessment platform with micro-services to serve
internal and public services and applications

3. Definitions
Apache Spark™ is a fast and general engine for large-scale
data processing.
Apache Spark is a cluster computing platform designed to
be fast and general-purpose.
Spark Streaming makes it easy to build scalable fault-tolerant
streaming applications.
Tachyon is a memory-centric distributed file system
enabling reliable file sharing at memory-speed across
cluster frameworks

4. Spark Stack

5. Stack with Tachyon

6. Distributed Execution

7. HDFS Architecture

8. HDFS Block Replication

9. Limitations of Map Reduce

10. Spark Runtime

11. RDD is an Interface
Advanced Spark Internals and Tuning

12. Sample Application

13. Narrow & Wide Dependencies

14. Tachyon System Architecture
Tachyon: Memory Throughput I/O for Cluster Computing Frameworks

15. Spark Execution Plan

16. Spark Executor on Worker Node

17. Fault-Tolerant in Spark Streaming
Could data be lost if the receiving node crashes before it replicates
incoming data to other data node(s)?
It happens. Ooyala loses 1% of their data but it is considered as
acceptable.
What can we do to prevent data loss?
We could persist events before they reach Spark Streaming Receiver,
replay the events/messages after receiver crashes and recovers.

18. Spark Internal Operations

19. Data Locality