Spark is a general-purpose cluster computing framework that provides APIs in Java, Scala, and Python. It supports distributed computing using Resilient Distributed Datasets (RDDs) and can handle large-scale data processing faster than Hadoop. Spark also supports SQL queries, streaming data, machine learning algorithms, and graph processing. RDDs are immutable, partitioned collections of elements that can be operated on in parallel. They provide a fault-tolerant abstraction of data across clusters.

1. $> spark-start
[ Kafka and Spark streaming ]

3. cc : https://www.ku.ac.th/scc2009/SCC2009_advance.pdf
Functional parallelization Data parallelization
E = (A + B) * (C + D)
CPU CPU
(C + D)(A + B)
CPU CPU
Input = [1,123,512,46]
1 * 2 123 * 2
E = ( A * 2)

4. MapReduce

5. Key Feature
● General propose cluster for large scale data processing.
● Speed − 100x faster than Hadoop for Logistic Regression Algorithm.
● Supports multiple languages − Spark provides built-in APIs in Java, Scala, or Python.
● Advanced Analytics − Spark not only supports ‘Map’ and ‘Reduce’. It also supports
SQL queries, Streaming data, Machine learning (ML), and Graph algorithms.

6. Spark Framework

7. Spark SQL
cc :
https://subscription.packtpub.com/book/big_data_and_business_intellige
nce/9781785884696/4/ch04lvl1sec26/architecture-of-spark-sql

8. Spark Streaming
cc : https://databricks.com/glossary/what-is-spark-streaming

9. Spark MLlib
cc : https://www.youtube.com/watch?v=DBxcua0Vmvk

10. Spark GraphX
cc : https://www.usenix.org/sites/default/files/conference/protected-files/osdi14_slides_gonzalez.pdf

11. Spark Cluster
cc : https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-settings

13. cc : https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html

14. Hadoop
cc : https://www.tutorialspoint.com/apache_spark/apache_spark_rdd.htm
Spark

15. cc : https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd.html#lineage
Resilient
Distributed
Dataset
Fault-tolerant with the help of RDD lineage graph
Data across multiple node
A collection of partitioned data

16. cc : https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd.html#lineage
● In-Memory stored in memory as much (size) and long (time) as possible.
● Immutable or Read-Only, i.e. it does not change once created and can only be transformed using
transformations to new RDDs.
● Lazy evaluated, i.e. the data inside RDD is not available or transformed until an action is executed that
triggers the execution.
● Cacheable, i.e. you can hold all the data in a persistent "storage" like memory (default and the most
preferred) or disk (the least preferred due to access speed).
● Parallel, i.e. process data in parallel.
● Typed — RDD records have types, e.g. Long in RDD[Long] or (Int, String) in RDD[(Int, String)].
● Partitioned — records are partitioned (split into logical partitions) and distributed across nodes in a cluster.
● Location-Stickiness — RDD can define placement preferences to compute partitions (as close to the
records as possible).

17. cc : https://www.youtube.com/watch?v=x8xXXqvhZq8

18. cc : https://www.youtube.com/watch?v=x8xXXqvhZq8

19. cc : https://www.youtube.com/watch?v=x8xXXqvhZq8

20. cc : https://www.youtube.com/watch?v=x8xXXqvhZq8

22. Zeppelin

23. ● Simple producer https://github.com/Aorjoa/kafka-simple-producer
● Simple Spark consumer https://github.com/Aorjoa/kafka-simple-consumer
● Spark Streaming twitter consumer https://github.com/Aorjoa/kafka-twitter-comsumer
● Spark Streaming twitter producer https://github.com/Aorjoa/kafka-simple-producer