Spark is an in-memory cluster computing framework that can access data from HDFS. It uses Resilient Distributed Datasets (RDDs) as its fundamental data structure. RDDs support transformations that create new datasets and actions that return values. DataFrames are equivalent to relational tables that allow for optimizations. HiveContext allows Spark to query data stored in Hive. Queries can be written using HiveQL, which is converted to Spark jobs.

1. A Step to programming withA Step to programming with
Rahul Kumar
Trainee - Software Consultant
Knoldus Software LLP
Rahul Kumar
Trainee - Software Consultant
Knoldus Software LLP

2. Building Spark :
1. Pre Build Spark
http://mirror.fibergrid.in/apache/spark/spark-1.6.2/spark-1.6.2-bin-hadoop2.6.tgz
2. Source Code
http://mirror.fibergrid.in/apache/spark/spark-1.6.2/spark-1.6.2.tgz
Goto the SPARK_HOME directory.
Execute : mvn -Dhadoop.version=1.2.1 -Phadoop-1 -DskipTests clean package
To start spark
goto the SPARK_HOME/bin
Execute ./spark-shell

3. ● The main feature of Spark is its in-memory cluster
computing that increases the processing speed of an
application.
● Spark is not a modified version of Hadoop because
it has its own cluster management.
● Spark uses Hadoop in two ways – one is storage
and second is processing. Since Spark has its own
cluster management computation, it uses Hadoop
for storage purpose only.
Spark Features :

4. Spark applications run as independent
sets of processes on a cluster,coordinated
by the SparkContext object in your main
program (called the driver program).

7. ● Resilient Distributed Datasets (RDD) is a fundamental
data structure of Spark.
● It is an immutable distributed collection of objects.
● RDDs can contain any type of Python, Java, or Scala
objects, including user-defined classes.
● There are two ways to create RDDs: parallelizing an
existing collection in your driver program
● e.g. val data = Array(1, 2, 3, 4, 5)
val distData = sc.parallelize(data)
● val distFile = sc.textFile("data.txt")
distFile: RDD[String] = MappedRDD@1d4cee08
RDD :

8. ● RDD(SPARK)
● HDFS(HADOOP)

9. ● RDDs support two types of operations:
✔ Transformations, which create a new dataset from an existing one, and
✔Actions, which return a value to the driver program after running a
computation on the dataset.
●For example,
✔ map is a transformation that passes each dataset element through a
function and returns a new RDD representing the results.
✔ reduce is an action that aggregates all the elements of the RDD using
some function and returns the final result to the driver program
●All transformations in Spark are lazy, in that they do not
compute their results right away
● RDDs support two types of operations:
✔ Transformations, which create a new dataset from an existing one, and
✔Actions, which return a value to the driver program after running a
computation on the dataset.
●For example,
✔ map is a transformation that passes each dataset element through a
function and returns a new RDD representing the results.
✔ reduce is an action that aggregates all the elements of the RDD using
some function and returns the final result to the driver program
●All transformations in Spark are lazy, in that they do not
compute their results right away
RDD :

10. ● A DataFrame is equivalent to a relational table
in Spark SQL.
DataFrame :
● Steps to create DataFrame :
 Create SparkContext object :
– val conf = new
SparkConf().setAppName("Demo").setMaster("local[2]")
– val sc = new SparkContext(conf)
 Create SqlContext object :
– val sqlContext = new SQLContext(sc)
 Read Data From Files :
– val df = sqlContext.read.json("src/main/scala/emp.json")

11. ● A data frame is a table, or two-dimensional array-like structure, in which each column
contains measurements on one variable, and each row contains one case.
● DataFrame has additional metadata due to its tabular format, which allows Spark to
run certain optimizations on the finalized query.
● An RDD, on the other hand, is merely a Resilient Distributed Dataset
that is more of a blackbox of data that cannot be optimized as the
operations that can be performed against it are not as constrained.
● However, you can go from a DataFrame to an RDD via its rdd
method, and you can go from an RDD to a DataFrame (if the RDD is
in a tabular format) via the toDF method
DataFrame and RDD :

12. DataFrame Transformations :
● Def orderBy(sortExprs: Column*): DataFrame
● Def select(cols: Column*): DataFrame
● Def show(): Unit
● Def filter(conditionExpr: String): DataFrame
● Def groupBy(cols: Column*): GroupedData

13. ●Def collect(): Array[Row]
●Def collectAsList(): List[Row]
●Def count(): Long
●Def head(): Row
●Def head(n: Int): Array[Row]
●Def collect(): Array[Row]
●Def collectAsList(): List[Row]
●Def count(): Long
●Def head(): Row
●Def head(n: Int): Array[Row]
DataFrame Actions :

14. ● Hive is a data warehouse infrastructure tool to process structured data
in Hadoop.
● It resides on top of Hadoop to summarize Big Data, and makes
querying and analyzing easy.
● It stores schema in a database and processed data into HDFS.
● It provides SQL type language for querying called HiveQL or HQL.
● It is designed for OLAP.
Hive :

15. ● Hive comes bundled with the Spark library as
HiveContext, which inherits from SQLContext.
● Using HiveContext, you can create and find tables in
the HiveMetaStore and write queries on it using
HiveQL.
● Users who do not have an existing Hive deployment
can still create a HiveContext.
● When not configured by the hive-site.xml, the context
automatically creates a metastore called metastore_db
and a folder called warehouse in the current directory.
Spark-Hive :

16. ➢ Spark SQL supports queries written using HiveQL.
➢ Its a SQL-like language that produces queries that are
converted to Spark jobs.
➢ HiveQL is more mature and supports more complex
queries than Spark SQL.
Spark-Hive :(continued)

17. 1) first create a SqlContext instance,
val sqlContext = new SqlContext(sc)
2) submit the queries by calling the sql method on the HiveContext instance.
val res=sqlContext.sql("select * from employee")
To construct a HiveQL query,
1) first create a new HiveContext instance,
val conf = new SparkConf().setAppName("Demo").setMaster("local[2]")
val sc = new SparkContext(conf)
val hiveContext = new HiveContext(sc)
2) submit the queries by calling the sql method on the HiveContext instance.
val res=hiveContext.sql("select * from employee")

19. ReferencesReferences
● http://spark.apache.org/docs/latest/sql-programming-guide.html
● http://www.tutorialspoint.com/spark_sql/spark_introduction.htm
● https://cwiki.apache.org