Apache Spark is a fast and general-purpose cluster computing system. It is compatible with Apache Hadoop but can be up to 100x faster. Spark uses Resilient Distributed Datasets (RDDs) that can be operated on in parallel. Key concepts include transformations that create new RDDs from existing ones, and actions that return values to the driver program. Spark is written in Scala and supports Java, Scala, and Python APIs. It has a growing ecosystem of tools including Spark Streaming, MLlib, and GraphX.

1. APACHE-SPARK
LARGE-SCALE DATA PROCESSING ENGINE
Bartosz Bogacki <bbogacki@bidlab.pl>

2. CTO, CODER, ROCK CLIMBER
• current:
• Chief Technology Officer at Bidlab
• previous:
• IT Director at
InternetowyKantor.pl SA
• Software Architect / Project
Manager at Wolters Kluwer
Polska
• find out more (if you care):
• linkedin.com/in/bartoszbogacki

3. WE PROCESS MORETHAN
200GB OF LOGS DAILY
Did I mention that…
?

4. WHY?
• To discover inventory and potential
• To optimize traffic
• To optimize campaigns
• To learn about trends
• To calculate conversions

5. APACHE SPARK !

6. HISTORY
• 2013-06-19 Project enters Apache incubation
• 2014-02-19 Project established as an ApacheTop
Level Project.
• 2014-05-30 Spark 1.0.0 released

7. • "Apache Spark is a (lightning-) fast and
general-purpose cluster computing system"
• Engine compatible with Apache Hadoop
• Up to 100x faster than Hadoop
• Less code to write, more elastic
• Active community (117 developers
contributed to release 1.0.0)

8. KEY CONCEPTS
• Spark /YARN / Mesos resources compatible
• HDFS / S3 support built-in
• RDD - Resilient Distribiuted Dataset
• Transformations & Actions
• Written in Scala,API for Java / Scala / Python

9. ECOSYSTEM
• Spark Streaming
• Shark
• MLlib (machine learning)
• GraphX
• Spark SQL

10. RDD
• Collections of objects
• Stored in memory (or disk)
• Spread across the cluster
• Auto-rebuild on failure

11. TRANSFORMATIONS
• map / flatMap
• filter
• union / intersection / join / cogroup
• distinct
• many more…

12. ACTIONS
• reduce / reduceByKey
• foreach
• count / countByKey
• first / take / takeOrdered
• collect / saveAsTextFile / saveAsObjectFile

13. EXAMPLES
val s1=sc.parallelize(Array(1,2,3,4,5))
val s2=sc.parallelize(Array(3,4,6,7,8))
val s3=sc.parallelize(Array(1,2,2,3,3,3))
!
s2.map(num => num * num)
// => 9, 16, 36, 49, 64
s1.reduce((a,b) => a + b)
// => 15
s1 union s2
// => 1, 2, 3, 4, 5, 3, 4, 6, 7, 8
s1 subtract s2
// => 1, 5, 2
s1 intersection s2
// => 4, 3
s3.distinct
// => 1, 2, 3

14. EXAMPLES
val set1 = sc.parallelize(Array[(Integer,String)]
((1,”bartek"),(2,"jacek"),(3,"tomek")))
val set2 = sc.parallelize(Array[(Integer,String)]
((2,”nowak”),(4,"kowalski"),(5,"iksiński")))
!
set1 join set2
// =>(2,(jacek,nowak))
set1 leftOuterJoin set2
// =>(1,(bartek,None)), (2,(jacek,Some(nowak))), (3,
(tomek,None))
set1 rightOuterJoin set2
// =>(4,(None,kowalski)), (5,(None,iksiński)), (2,
(Some(jacek),nowak))

15. EXAMPLES
set1.cogroup(set2).sortByKey()
// => (1,(ArrayBuffer(bartek),ArrayBuffer())), (2,
(ArrayBuffer(jacek),ArrayBuffer(nowak))), (3,
(ArrayBuffer(tomek),ArrayBuffer())), (4,
(ArrayBuffer(),ArrayBuffer(kowalski))), (5,
(ArrayBuffer(),ArrayBuffer(iksiński)))
!
set2.map((t) => (t._1, t._2.length))
// => (2,5), (4,8), (5,8)
!
val set3 = sc.parallelize(Array[(String,Long)]
(("onet.pl",1), ("onet.pl",1), ("wp.pl",1))
!
set3.reduceByKey((n1,n2) => n1 + n2)
// => (onet.pl,2), (wp.pl,1)

16. HANDS ON

17. RUNNING EC2
SPARK CLUSTER
./spark-ec2 -k spark-key -i spark-key.pem
-s 5
-t m3.2xlarge
launch cluster-name
--region=eu-west-1

18. SPARK CONSOLE

19. LINKING WITH SPARK
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.0.0</version>
</dependency>
If you want to use HDFS
!
groupId = org.apache.hadoop
artifactId = hadoop-client
version = <your-hdfs-version>
If you want to use Spark Streaming
!
groupId = org.apache.spark
artifactId = spark-streaming_2.10
version = 1.0.0

20. INITIALIZING
• SparkConf conf = new SparkConf()
.setAppName("TEST")
.setMaster("local");
• JavaSparkContext sc = new
JavaSparkContext(conf);

21. CREATING RDD
• List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);
• JavaRDD<Integer> distData = sc.parallelize(data);

22. CREATING RDD
• JavaRDD<String> logLines = sc.textFile("data.txt");

23. CREATING RDD
• JavaRDD<String> logLines = sc.textFile(”hdfs://
<HOST>:<PORT>/daily/data-20-00.txt”);
• JavaRDD<String> logLines = sc.textFile(”s3n://my-
bucket/daily/data-*.txt”);

24. TRANSFORM
JavaRDD<Log> logs =
logLines.map(new Function<String, Log>() {
public Log call(String s) {
return LogParser.parse(s);
}
}).filter(new Function<Log, Boolean>(){
public Integer call(Log log) {
return log.getLevel() == 1;
}
});

25. ACTION :)
logs.count();

26. TRANSFORM-ACTION
List<Tuple2<String,Integer>> result =
sc.textFile(”/data/notifies-20-00.txt”)
.mapToPair(new PairFunction<String, String, Integer>() {
@Override
public Tuple2<String, Integer> call(String line) throws Exception {
NotifyRequest nr = LogParser.parseNotifyRequest(line);
return new Tuple2<String, Integer>(nr.getFlightId(), 1);
}
})
.reduceByKey(new Function2<Integer, Integer, Integer>(){
@Override
public Integer call(Integer v1, Integer v2) throws Exception {
return v1 + v2;
}})
.sortByKey()
.collect();

27. FUNCTIONS,
PAIRFUNCTIONS,
ETC.

28. BROADCASTVARIABLES
• "allow the programmer to keep a read-only
variable cached on each machine rather than
shipping a copy of it with tasks"
Broadcast<int[]> broadcastVar =
sc.broadcast(new int[] {1, 2, 3});
!
broadcastVar.value();
// returns [1, 2, 3]

29. ACCUMULATORS
• variables that are only “added” to through an associative
operation (add())
• only the driver program can read the accumulator’s value
Accumulator<Integer> accum = sc.accumulator(0);
!
sc.parallelize(Arrays.asList(1, 2, 3, 4)).foreach(x ->
accum.add(x));
!
accum.value();
// returns 10

30. SERIALIZATION
• All objects used in your code have to be
serializable
• Otherwise:
org.apache.spark.SparkException: Job aborted: Task not
serializable: java.io.NotSerializableException

31. USE KRYO SERIALIZER
public class MyRegistrator implements KryoRegistrator {
@Override
public void registerClasses(Kryo kryo) {
kryo.register(BidRequest.class);
kryo.register(NotifyRequest.class);
kryo.register(Event.class);
}
}
sparkConfig.set(
"spark.serializer", "org.apache.spark.serializer.KryoSerializer");
sparkConfig.set(
"spark.kryo.registrator", "pl.instream.dsp.offline.MyRegistrator");
sparkConfig.set(
"spark.kryoserializer.buffer.mb", "10");

32. CACHE !
JavaPairRDD<String, Integer> cachedSet =
sc.textFile(”/data/notifies-20-00.txt”)
.mapToPair(new PairFunction<String, String, Integer>() {
@Override
public Tuple2<String, Integer> call(String line) throws Exception
{
NotifyRequest nr = LogParser.parseNotifyRequest(line);
return new Tuple2<String, Integer>(nr.getFlightId(), 1);
}
}).cache();

33. RDD PERSISTANCE
• MEMORY_ONLY
• MEMORY_AND_DISK
• MEMORY_ONLY_SER
• MEMORY_AND_DISK_SER
• DISK_ONLY
• MEMORY_ONLY_2, MEMORY_AND_DISK_2, …
• OFF_HEAP (Tachyon, ecperimental)

34. PARTITIONS
• RDD is partitioned
• You may (and probably should) control number
and size of partitions with coalesce() method
• By default 1 input file = 1 partition

35. PARTITIONS
• If your partitions are too big, you’ll face:
[GC 5208198K(5208832K), 0,2403780 secs]
[Full GC 5208831K->5208212K(5208832K), 9,8765730 secs]
[Full GC 5208829K->5208238K(5208832K), 9,7567820 secs]
[Full GC 5208829K->5208295K(5208832K), 9,7629460 secs]
[GC 5208301K(5208832K), 0,2403480 secs]
[Full GC 5208831K->5208344K(5208832K), 9,7497710 secs]
[Full GC 5208829K->5208366K(5208832K), 9,7542880 secs]
[Full GC 5208831K->5208415K(5208832K), 9,7574860 secs]
WARN storage.BlockManagerMasterActor: Removing
BlockManager BlockManagerId(0, ip-xx-xx-xxx-xxx.eu-
west-1.compute.internal, 60048, 0) with no recent heart
beats: 64828ms exceeds 45000ms

36. RESULTS
• result.saveAsTextFile(„hdfs://<HOST>:<PORT>/
out.txt")
• result.saveAsObjectFile(„/result/out.obj”)
• collect()

37. PROCESS RESULTS
PARTITION BY PARTITION
for (Partition partition : result.rdd().partitions()) {
List<String> subresult[] =
result.collectPartitions(new int[] { partition.index() });
for (String line : subresult[0])
{
System.out.println(line);
}
}

38. SPARK STREAMING

39. „SPARK STREAMING IS AN EXTENSION OFTHE
CORE SPARK APITHAT ENABLES
HIGH-THROUGHPUT, FAULT-TOLERANT
STREAM PROCESSING OF LIVE DATA STREAMS.”

40. HOW IT WORKS?

41. DSTREAMS
• continuous stream of data, either the input data
stream received from source, or the processed
data stream generated by transforming the input
stream
• represented by a continuous sequence of RDDs

42. INITIALIZING
• SparkConf conf = new
SparkConf().setAppName("Real-Time
Analytics").setMaster("local");
• JavaStreamingContext jssc = new
JavaStreamingContext(conf, new
Duration(TIME_IN_MILIS));;

43. CREATING DSTREAM
• JavaReceiverInputDStream<String> logLines =
jssc.socketTextStream(sourceAddr, sourcePort,
StorageLevels.MEMORY_AND_DISK_SER);

44. DATA SOURCES
• plainTCP sockets
• Apache Kafka
• Apache Flume
• ZeroMQ

45. TRANSFORMATIONS
• map, flatMap, filter, union, join, etc.
• transform
• updateStateByKey

46. WINDOW OPERATIONS
• window
• countByWindow / countByValueAndWindow
• reduceByWindow / reduceByKeyAndWindow

47. OUTPUT OPERTIONS
• print
• foreachRDD
• saveAsObjectFiles
• saveAsTextFiles
• saveAsTextFiles

48. THINGSTO REMEMBER

49. USE SPARK-SHELLTO LEARN

50. PROVIDE ENOUGH RAM
TO WORKERS

51. PROVIDE ENOUGH RAM
TO EXECUTOR

52. SET FRAME SIZE / BUFFERS
ACCORDINGLY

53. USE KRYO SERIALIZER

54. SPLIT DATATO APPROPRIATE
NUMBER OF PARTITIONS

55. PACKAGEYOUR APPLICATION
IN UBER-JAR

56. DESIGNYOUR DATA FLOW
AND…

57. BUILD A FRAMEWORKTO
PROCESS DATA EFFICIENTLY

58. IT’S EASIER WITH SCALA!
// word count example
inputLine.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _);

59. HOW WE USE SPARK?

60. HOW WE USE SPARK?

61. HOW WE USE SPARK?

62. THANKS!

63. we’re hiring !
mail me: bbogacki@bidlab.pl