** PySpark Certification Training: https://www.edureka.co/pyspark-certification-training**
This Edureka tutorial on PySpark Tutorial will provide you with a detailed and comprehensive knowledge of Pyspark, how it works, the reason why python works best with Apache Spark. You will also learn about RDDs, data frames and mllib.
6. www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
PySpark
Spark is an open-source cluster-computing framework which is built around speed, ease of use,
and streaming analytics
Python is general purpose high level programming language. It provides wide range of libraries
and is majorly used for Machine Learning and Data Science
• It is a Python API for Spark majorly used for Data Science and Analysis
• Using PySpark, you can work with Spark RDDs in Python
20. www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Spark Context
Spark Context
Spark
Context
Py Process
Py4J
Worker (JVM)
Block 1
Worker(JVM)
Block 2
Local FS
Py Process
Py Process
Py Process
Local Cluster
SparkContext is the entry point to any spark functionality
Socket
Socket
Pipe
Pipe
Pipe
Pipe
23. www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
PySpark
Basic life cycle of a PySpark program
01 03
02
Create RDDs Cache RDDs
Lazy
Transformation
Create RDDs from some external
data source or parallelize a
collection in your driver
program.
Lazily transform the base RDDs
into new RDDs using
transformations
Cache some of those RDDs for
future reuse
04 Perform Actions
Perform actions to execute
parallel computation and to
produce results
25. www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Resilient Distributed Dataset (RDDs)
RDDs is the building block of every Spark application and is immutable
R
D
D
esilient
istributed
ataset
Fault tolerant and is capable of rebuilding data on failure
Data is distributed among the multiple nodes in a cluster
Collection of partitioned data with primitive values or values of value
26. www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Transformations & Actions in RDDs
To work on this immutable data, you need to create a new one via Transformations and Actions
Transformations
❑ map
❑ flatMap
❑ filter
❑ distinct
❑ reduceByKey
❑ mapPartitions
❑ sortBy
Actions
❑ collect
❑ collectAsMap
❑ reduce
❑ countByKey/countByValue
❑ take
❑ first
28. www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Broadcast & Accumulator
Parallel processing is achieved in Spark by using shared variables
Shared Variables
Broadcast Accumulator
These variables are used to save
the copy of data across all
nodes
These variables are used to
aggregate the information
through associative and
commutative operations
30. www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
SparkConf
SparkConf provides the configurations to run a Spark application on a local system or a cluster
SparkConf object is used to set different parameters which takes priority over the system properties
Once SparkConf object is passed to Spark, it becomes immutable
31. www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
SparkConf
Attributes of SparkConf class
set(key, value)………………………………………
setMaster(value)……………………………………
setAppName(value)…………………………………
get(key, defaultValue=None)………
setSparkHome(value)……………………………
Sets Config property
Sets the master URL
Sets an application’s name
Gets the configuration value of a key
Sets the Spark installation path on worker nodes
33. www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
SparkFiles
SparkFiles class helps in resolving the paths of files added to the Spark
get(filename)……………………………………………
getrootdirectory()………………………………
It specifies the path of the file that is added through sc.addFile()
It specifies the path to the root directory of the file that is added through sc.addFile()