Knoldus organized a Meetup on 1 April 2015. In this Meetup, we introduced Spark with Scala. Apache Spark is a fast and general engine for large-scale data processing. Spark is used at a wide range of organizations to process large datasets.

1. Introduction to
Spark with Scala
Introduction to
Spark with Scala
Himanshu Gupta
Software Consultant
Knoldus Software LLP
Himanshu Gupta
Software Consultant
Knoldus Software LLP

2. Who am I ?Who am I ?
Himanshu Gupta (@himanshug735)
Software Consultant at Knoldus Software LLP
Spark & Scala enthusiast
Himanshu Gupta (@himanshug735)
Software Consultant at Knoldus Software LLP
Spark & Scala enthusiast

3. AgendaAgenda
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo

4. What is Apache Spark ?What is Apache Spark ?
Fast and general engine for large-scale data processing
with libraries for SQL, streaming, advanced analytics
Fast and general engine for large-scale data processing
with libraries for SQL, streaming, advanced analytics

5. Spark HistorySpark History
Project Begins
at
UCB AMP Lab
20092009
20102010
Open Sourced
Apache Incubator
20112011
20122012
20132013
20142014
20152015
Data Frames
Cloudera
Support
Apache
Top level
Spark
Summit
2013
Spark
Summit
2014

6. Spark StackSpark Stack
Img src - http://spark.apache.org/Img src - http://spark.apache.org/

7. Fastest Growing Open Source ProjectFastest Growing Open Source Project
Img src - https://databricks.com/blog/2015/03/31/spark-turns-five-years-old.htmlImg src - https://databricks.com/blog/2015/03/31/spark-turns-five-years-old.html

8. AgendaAgenda
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo

9. Code SizeCode Size
Img src - http://spark-summit.org/wp-content/uploads/2013/10/Zaharia-spark-summit-2013-matei.pdfImg src - http://spark-summit.org/wp-content/uploads/2013/10/Zaharia-spark-summit-2013-matei.pdf

10. Word Count Ex.
public static class WordCountMapClass extends MapReduceBase
implements Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
}
public void map(LongWritable key, Text value, OutputCollector<Text,
IntWritable> output, Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer itr = new StringTokenizer(line);
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
output.collect(word, one);
}
}
public static class WorkdCountReduce extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output, Reporter reporter) throws
IOException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
}
}
val file = spark.textFile("hdfs://...")
val counts = file.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")

11. Daytona GraySort Record:
Data to sort 100TB
Daytona GraySort Record:
Data to sort 100TB
Img src -http://www.slideshare.net/databricks/new-directions-for-apache-spark-in-2015Img src -http://www.slideshare.net/databricks/new-directions-for-apache-spark-in-2015
Hadoop (2013):Hadoop (2013): 2100 nodes2100 nodes
72 minutes72 minutes
Spark (2014):Spark (2014): 206 nodes206 nodes
23 minutes23 minutes

12. Runs EverywhereRuns Everywhere
Img src - http://spark.apache.org/

13. Who are using Apache Spark ?Who are using Apache Spark ?
Img src - http://www.slideshare.net/datamantra/introduction-to-apache-spark-45062010Img src - http://www.slideshare.net/datamantra/introduction-to-apache-spark-45062010

14. AgendaAgenda
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo

15. Brief Introduction to RDDBrief Introduction to RDD
 RDD stands for Resilient Distributed Dataset
 A fault tolerant, distributed collection of objects.
 In Spark all work is expressed in following ways:
1) Creating new RDD(s)
2) Transforming existing RDD(s)
3) Calling operations on RDD(s)
 RDD stands for Resilient Distributed Dataset
 A fault tolerant, distributed collection of objects.
 In Spark all work is expressed in following ways:
1) Creating new RDD(s)
2) Transforming existing RDD(s)
3) Calling operations on RDD(s)

16. Example (RDD)Example (RDD)
val master = "local"
val conf = new SparkConf().setMaster(master)
This is the Spark
Configuration

17. Example (RDD)Example (RDD)
val master = "local"
val conf = new SparkConf().setMaster(master)
val sc = new SparkContext(conf)
This is the Spark
Context
Contd...Contd...

18. Example (RDD)Example (RDD)
val master = "local"
val conf = new SparkConf().setMaster(master)
val sc = new SparkContext(conf)
This is the Spark
Context
Contd...Contd...

19. Example (RDD)Example (RDD)
val master = "local"
val conf = new SparkConf().setMaster(master)
val sc = new SparkContext(conf)
val lines = sc.textFile("data.txt")
Extract lines
from text file
Contd...Contd...

20. Example (RDD)Example (RDD)
val master = "local"
val conf = new SparkConf().setMaster(master)
val sc = new SparkContext(conf)
val lines = sc.textFile("demo.txt")
val words = lines.flatMap(_.split(" ")).map((_,1))
Map lines
to words
map
Contd...Contd...

21. Example (RDD)Example (RDD)
val master = "local"
val conf = new SparkConf().setMaster(master)
val sc = new SparkContext(conf)
val lines = sc.textFile("demo.txt")
val words = lines.flatMap(_.split(" ")).map((_,1))
val wordCountRDD = words.reduceByKey(_ + _)
Word Count RDD
map groupBy
Contd...Contd...

22. Example (RDD)Example (RDD)
val master = "local"
val conf = new SparkConf().setMaster(master)
val sc = new SparkContext(conf)
val lines = sc.textFile("demo.txt")
val words = lines.flatMap(_.split(" ")).map((_,1))
val wordCountRDD = words.reduceByKey(_ + _)
val wordCount = wordCountRDD.collect
Map[word, count] map groupBy
collect
Starts
Computation
Contd...Contd...

23. Example (RDD)Example (RDD)
val master = "local"
val conf = new SparkConf().setMaster(master)
val sc = new SparkContext(conf)
val lines = sc.textFile("demo.txt")
val words = lines.flatMap(_.split(" ")).map((_,1))
val wordCountRDD = words.reduceByKey(_ + _)
val wordCount = wordCountRDD.collect
map groupBy
collect
Transformation Action
Contd...Contd...

24. AgendaAgenda
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo

25. Brief Introduction to Spark StreamingBrief Introduction to Spark Streaming
Img src - http://spark.apache.org/Img src - http://spark.apache.org/

26. How Spark Streaming Works ?How Spark Streaming Works ?
Img src - http://spark.apache.org/Img src - http://spark.apache.org/

27. Why we need Spark Streaming ?Why we need Spark Streaming ?
High Level API:High Level API:
TwitterUtils.createStream(...)
.filter(_.getText.contains("Spark"))
.countByWindow(Seconds(10), Seconds(5))
//Counting tweets on a sliding window
Fault Tolerant:Fault Tolerant:
Integration:Integration:
Img src - http://spark.apache.org/Img src - http://spark.apache.org/
Integrated with Spark SQL, MLLib,
GraphX...

28. Example (Spark Streaming)Example (Spark Streaming)
val master = "local"
val conf = new SparkConf().setMaster(master)
Specify Spark
Configuration

29. Example (Spark Streaming)Example (Spark Streaming)
val master = "local"
val conf = new SparkConf().setMaster(master)
val ssc = new StreamingContext(conf, Seconds(10))
Setup Stream
Context
Contd...Contd...

30. Example (Spark Streaming)Example (Spark Streaming)
val master = "local"
val conf = new SparkConf().setMaster(master)
val ssc = new StreamingContext(conf, Seconds(10))
val lines = ssc.socketTextStream("localhost", 9999)
This is the
ReceiverInputDStream
lines
DStream
at time
0 - 1
at time
1 - 2
at time
2 - 3
at time
3 - 4
Contd...Contd...

31. Example (Spark Streaming)Example (Spark Streaming)
val master = "local"
val conf = new SparkConf().setMaster(master)
val ssc = new StreamingContext(conf, Seconds(10))
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" ")).map((_, 1))
lines
DStream
at time
0 - 1
words/pairs
DStream
at time
1 - 2
at time
2 - 3
at time
3 - 4
map
Creates a Dstream
(sequence of RDDs)
Contd...Contd...

32. Example (Spark Streaming)Example (Spark Streaming)
val master = "local"
val conf = new SparkConf().setMaster(master)
val ssc = new StreamingContext(conf, Seconds(10))
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" ")).map((_, 1))
val wordCounts = words.reduceByKey(_ + _)
lines
DStream
at time
0 - 1
words/pairs
DStream
at time
1 - 2
at time
2 - 3
at time
3 - 4
wordCount
DStream
map
groupBy
Groups Dstream
by Words
Contd...Contd...

33. Example (Spark Streaming)Example (Spark Streaming)
val master = "local"
val conf = new SparkConf().setMaster(master)
val ssc = new StreamingContext(conf, Seconds(10))
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" ")).map((_, 1))
val wordCounts = words.reduceByKey(_ + _)
ssc.start()
lines
DStream
at time
0 - 1
words/pairs
DStream
at time
1 - 2
at time
2 - 3
at time
3 - 4
wordCount
DStream
map
groupBy
Start streaming
& computation
Contd...Contd...

34. AgendaAgenda
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo

35. How to Install Spark ?
 Download Spark from -
http://spark.apache.org/downloads.html
 Extract it to a suitable directory.
 Go to the directory via terminal & run following command -
mvn -DskipTests clean package
 Now Spark is ready to run in Interactive mode
./bin/spark-shell
 Download Spark from -
http://spark.apache.org/downloads.html
 Extract it to a suitable directory.
 Go to the directory via terminal & run following command -
mvn -DskipTests clean package
 Now Spark is ready to run in Interactive mode
./bin/spark-shell

36. sbt Setup
name := "Spark Demo"
version := "1.0"
scalaVersion := "2.10.5"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.2.1",
"org.apache.spark" %% "spark-streaming" % "1.2.1",
"org.apache.spark" %% "spark-sql" % "1.2.1",
"org.apache.spark" %% "spark-mllib" % "1.2.1"
)

37. AgendaAgenda
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo
● What is Spark ?
● Why we need Spark ?
● Brief introduction to RDD
● Brief introduction to Spark Streaming
● How to install Spark ?
● Demo

38. Demo

39. Download Code
https://github.com/knoldus/spark-scala

40. References
http://spark.apache.org/
http://spark-summit.org/2014
http://spark.apache.org/docs/latest/quick-start.html
http://stackoverflow.com/questions/tagged/apache-spark
https://www.youtube.com/results?search_query=apache+spark
http://apache-spark-user-list.1001560.n3.nabble.com/
http://www.slideshare.net/paulszulc/apache-spark-101-in-50-min

41. Presenter:
himanshu@knoldus.com
@himanshug735
Presenter:
himanshu@knoldus.com
@himanshug735
Organizer:
@Knolspeak
http://www.knoldus.com
http://blog.knoldus.com
Organizer:
@Knolspeak
http://www.knoldus.com
http://blog.knoldus.com
Thanks