Learn how to Install Spark. Check it out on the following link- https://www.experfy.com/training/courses/apache-spark-sql

1. MODULE 2
Installing Spark

2. • High-level physical cluster architecture
• Software architecture of a standalone cluster
• Installing Spark standalone locally
• Running the Spark Shell
• Running in python shell and ipython shell
• Running sample Spark code
• Using the Spark Session and Spark Context
• Creating a parallelized collection
Installing Spark
Module 2. Installing Spark 2
What we’ll cover:

3. Spark Cluster Components
Module 2. Installing Spark 3
Driver Program
Spark Session
● Spark Context
Worker Node
Executor
● Cache
● Task
● Task
● ...
Cluster Manager
Worker Node
Executor
● Cache
● Task
● Task
● ...

4. • Driver Node on the Cluster Manager, aka Master Node
• Driver Node in a Spark Cluster
• Driver Program
• Driver Process
“Driver”
4Module 2. Installing Spark

5. • Cluster Mode
o submit application to cluster manager
• Client Mode
o run the driver outside the cluster, with executors in the cluster
• Standalone Mode
o everything is on one machine
Execution Modes
Module 3. Spark Computing Framework 5

6. Standalone Spark Cluster
Module 2. Installing Spark 6
Driver Program
Spark Session
● Spark Context
Worker Node
Executor
● Cache
● Task
● Task
● ...
Cluster Manager

7. • Ensure that you have Java installed, and verify your version
• Most recent version you can install with Java 7: version 2.1.x
• If you have Java 8+ installed you can install version 2.2.x
Installing Local Standalone
Module 2. Installing Spark 7

8. • Work within a virtual environment or container
• If you are already familiar with this you can skip the next three slides
Recommendation for Success
Module 2. Installing Spark 8

9. • This is not required but is strongly recommended.
• a virtual environment creates an isolated environment for a project
o each project can have its own dependencies and not affect other projects
o you can easily and safely remove this environment when you are done with it
• Examples include Virtualenv, Anaconda, but there are many to good options
• You may also use a container-based environment, such as Docker.
• Use your preferred approach or organization’s recommended practice
Creating a virtual environment
Module 2. Installing Spark 9

10. Example : creating a virtual environment.
Anaconda
$ conda create -n my-env python
$ source activate my-env
# Here is the same thing, but specifying Python 2.7
$ conda create -n my-env python=2.7
$ source activate my-env
# Here is the same thing, but with Python 3.6
$ conda create -n my-env python=3.6
$ source activate my-env
Module 2. Installing Spark 10

11. Example : creating a virtual environment.
Virtualenv
$ cd /path/to/my/venvs
$ virtualenv ./my_venv
$ source ./my_env/bin/activate
Module 2. Installing Spark 11

12. • The most recent version of Pyspark you can install with Java 7 is version 2.1.x
• If you have Java 8+ installed you can install Pyspark 2.2.x
Install Java
Module 2. Installing Spark 12

13. The simple way to install Pyspark
Recommended for Spark 2.2.x onwards
$ pip install pyspark
Module 2. Installing Spark 13

14. • Download source and Build
o Browse to https://spark.apache.org/downloads.html
o Browse https://spark.apache.org/downloads.html
o Choose the following settings :
Installing Pyspark 2.1.2
Module 2. Installing Spark 14
o Click on spark-2.1.2.tgz

15. Installing Pyspark 2.1.2
Module 2. Installing Spark 15
o Save this to /ext/spark

16. Installing Pyspark 2.1.2
Recommended for following along in this course
$ curl http://apache.mirrors.lucidnetworks.net/spark/spark-2.1.2/spark-2.1.2.tgz -o spark-
2.1.2.tgz
# Confirm that the file size is as expected (~13MiB)
$ ls -lh spark-2.1.2.tgz
# Extract the contents
$ tar -xf spark-2.1.2.tgz
$ cd /ext/spark/spark-2.1.2
Module 2. Installing Spark 16

17. • Set Java home environment variable
o This is necessary only if Java home is not already set.
Installing Pyspark 2.1.2
Module 2. Installing Spark 17

18. Installing Pyspark 2.1.2
Set $JAVA_HOME
$ echo $JAVA_HOME
# If above returns empty, then you’ll need to set it. On Mac OS, you can use result of :
$ echo `/usr/libexec/java_home`
like so :
$ export JAVA_HOME=`/usr/libexec/java_home`
Module 2. Installing Spark 18

19. • Get the build instructions
o It is good practice to vet these yourself before running.
o Understand what you are going to do before doing it
o Spark evolves rapidly, so know how to upgrade!
• Browse to : https://spark.apache.org/documentation.html
o Scroll down to your version, click into that link
o For our case, this is https://spark.apache.org/docs/2.1.2/building-spark.html
o “Building Spark 2.2.1 using Maven requires Maven 3.3.9 or newer and Java 7+.”
o “Note that support for Java 7 was removed as of Spark 2.2.0.”
• Now to build it
Installing Pyspark 2.1.2
Module 2. Installing Spark 19

20. Installing Pyspark 2.1.2
Build it
$ export MAVEN_OPTS="-Xmx2g -XX:ReservedCodeCacheSize=512m"
$ ./build/mvn -DskipTests clean package
[INFO] Total time: 11:11 min
$ sudo pip install py4j
Module 2. Installing Spark 20

21. Installing Pyspark 2.1.2
Pro tip : turn down overly verbose logs
$ cd /ext/spark/spark-2.1.2
$ cp conf/log4j.properties.template conf/log4j.properties
Module 2. Installing Spark 21

22. • Pro tip: ensure the following lines are in ~/.bash_profile
o export SPARK_HOME="/ext/spark/spark-2.1.2"
o export PATH=$SPARK_HOME/bin:$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.4-src.zip:$PATH
• If you altered ~/.bash_profile, remember to run the following:
o $ source ~/.bash_profile
Done Installing Spark
Module 2. Installing Spark 22

23. Running Pyspark Shell
$ pyspark
Welcome to
____ __
/ __/__ ___ _____/ /__
_ / _ / _ `/ __/ '_/
/__ / .__/_,_/_/ /_/_ version 2.1.2
/_/
SparkSession available as 'spark'.
>>>
Module 2. Installing Spark 23

24. Running Pyspark Shell
$ pyspark
Welcome to
____ __
/ __/__ ___ _____/ /__
_ / _ / _ `/ __/ '_/
/__ / .__/_,_/_/ /_/_ version 2.1.2
/_/
SparkSession available as 'spark'.
>>> spark
<pyspark.sql.session.SparkSession object at 0x1050f8d50>
>>>
Module 2. Installing Spark 24

25. Running Pyspark Shell
$ pyspark
Welcome to
____ __
/ __/__ ___ _____/ /__
_ / _ / _ `/ __/ '_/
/__ / .__/_,_/_/ /_/_ version 2.1.2
/_/
SparkSession available as 'spark'.
>>> spark
<pyspark.sql.session.SparkSession object at 0x1050f8d50>
>>>
Module 2. Installing Spark 25

26. Running Pyspark Shell
>>> sc = spark.sparkContext
>>> rdd = sc.parallelize(["Hello", "World"])
>>> print(rdd.count())
2
>>> print(rdd.collect())
['Hello', 'World']
>>> print(rdd.take(2))
['Hello', 'World']
Module 2. Installing Spark 26

27. Running Pyspark in python shell
$ python
>>>
Module 2. Installing Spark 27

28. Running Pyspark in python shell
We’ll simplify this later -- for now let’s ensure we’re using what we just installed
$ python
>>> import sys
>>> import os
>>> SPARK_HOME = '/ext/spark/spark-2.1.2'
>>> SPARK_PY4J = "python/lib/py4j-0.10.4-src.zip"
>>> sys.path.insert(0, os.path.join(SPARK_HOME, "python")) # precede pre-existing
>>> sys.path.insert(0, os.path.join(SPARK_HOME, SPARK_PY4J))
>>> os.environ["SPARK_HOME"] = SPARK_HOME
>>>
Module 2. Installing Spark 28

29. Running Pyspark in python shell
>>> from pyspark.sql import SparkSession
>>> spark = SparkSession
... .builder
... .appName("Python Spark SQL basic example")
... .config("spark.some.config.option", "some-value")
... .getOrCreate()
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
>>> spark
<pyspark.sql.session.SparkSession object at 0x1112e4650>
Module 2. Installing Spark 29

30. Running Pyspark in python shell
>>> sc = spark.sparkContext
>>> rdd = sc.parallelize(["Hello", "World"])
>>> print(rdd.count())
2
>>> print(rdd.collect())
['Hello', 'World']
>>> print(rdd.take(2))
['Hello', 'World']
Module 2. Installing Spark 30

31. Running Pyspark in ipython shell
Repeat the steps we did in the python shell :
$ ipython
In [1]: import os
...: import sys
...: SPARK_HOME = '/ext/spark/spark-2.1.2'
...: SPARK_PY4J = "python/lib/py4j-0.10.4-src.zip"
...: sys.path.insert(0, os.path.join(SPARK_HOME, "python"))
...: sys.path.insert(0, os.path.join(SPARK_HOME, SPARK_PY4J)) # must precede IDE's py4j
...:
Module 2. Installing Spark 31

32. Running Pyspark in ipython shell
In [2]: java_home = "/Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home"
...: os.environ["JAVA_HOME"] = java_home
...: os.environ["SPARK_HOME"] = SPARK_HOME
...: from pyspark.sql import SparkSession
...:
Module 2. Installing Spark 32

33. Running Pyspark in ipython shell
In [3]: spark = SparkSession
...: .builder
...: .appName("Python Spark SQL basic example")
...: .config("spark.some.config.option", "some-value")
...: .getOrCreate()
...:
Module 2. Installing Spark 33

34. Running Pyspark in ipython shell
In [4]: spark?
Type: SparkSession
String form: <pyspark.sql.session.SparkSession object at 0x1111f5f10>
File: /ext/spark/spark-2.1.2/python/pyspark/sql/session.py
Docstring:
The entry point to programming Spark with the Dataset and DataFrame API.
A SparkSession can be used create :class:`DataFrame`, register :class:`DataFrame` as
tables, execute SQL over tables, cache tables, and read parquet files.
To create a SparkSession, use the following builder pattern:
[...]
Module 2. Installing Spark 34

35. Running Pyspark in ipython shell
Running the example code that we ran previously
In [5]: sc = spark.sparkContext
...: rdd = sc.parallelize(["Hello", "World"])
...: print(rdd.count())
...: print(rdd.collect())
...: print(rdd.take(2))
...:
2
['Hello', 'World']
['Hello', 'World']
Module 2. Installing Spark 35

36. Running Pyspark in ipython shell
In [6]: sc?
Type: SparkContext
String form: <pyspark.context.SparkContext object at 0x1111702d0>
File: /ext/spark/spark-2.1.2/python/pyspark/context.py
Docstring:
Main entry point for Spark functionality. A SparkContext represents the
connection to a Spark cluster, and can be used to create L{RDD} and
broadcast variables on that cluster.
Init docstring:
Create a new SparkContext. At least the master and app name should be set,
either through the named parameters here or through C{conf}.
Module 2. Installing Spark 36

37. Running Pyspark in ipython shell
In [7]: rdd?
Type: RDD
String form: ParallelCollectionRDD[0] at parallelize at PythonRDD.scala:475
File: /ext/spark/spark-2.1.2/python/pyspark/rdd.py
Docstring:
A Resilient Distributed Dataset (RDD), the basic abstraction in Spark.
Represents an immutable, partitioned collection of elements that can be
operated on in parallel.
Module 2. Installing Spark 37

38. Running Pyspark in ipython shell
Tab completion
T
In [8]: rdd.
rdd.aggregate rdd.coalesce rdd.context rdd.countByValue
rdd.aggregateByKey rdd.cogroup rdd.count rdd.ctx
rdd.cache rdd.collect rdd.countApprox rdd.distinct
rdd.cartesian rdd.collectAsMap rdd.countApproxDistinct rdd.filter
rdd.checkpoint rdd.combineByKey rdd.countByKey rdd.first
Module 2. Installing Spark 38

39. Running Pyspark in ipython shell
Magic functions
T
In [9]: %whos
Variable Type Data/Info
----------------------------------------
SPARK_HOME str /ext/spark/spark-2.1.2
SPARK_PY4J str python/lib/py4j-0.10.4-src.zip
SparkSession type <class 'pyspark.sql.session.SparkSession'>
java_home str /Library/Java/JavaVirtual<...>.7.0_80.jdk/Contents/Home
rdd RDD ParallelCollectionRDD[0] <...>ze at PythonRDD.scala:475
sc SparkContext <pyspark.context.SparkCon<...>xt object at 0x1111702d0>
spark SparkSession <pyspark.sql.session.Spar<...>on object at 0x1111f5f10>
Module 2. Installing Spark 39

40. • We learned several ways of installing Pyspark
• We ran the Spark Shell
• We showed how to run our new install in python and ipython
• We tested Pyspark by running sample code in all three of these shells
• We used two important objects: the Spark Session, and the Spark Context
• We created a RDD and inspected its contents.
What we covered
Module 2. Installing Spark 40

41. • Next time we’ll cover more ways of running Spark.
• We’ll show it in an IDE, and notebook.
• We’ll get our first view of the Spark UI
Next time: Running Spark
Module 2. Installing Spark 41