The presentation of Apache Spark by Mylène Reiners during our first Eindhoven Java Meetup (see http://www.opencirclesolutions.nl/eindhoven-java-meetup/).

1. Apache Spark
AN ENGINE FOR LARGE-SCALE DATA PROCESSING

2. Introducing myself…
• Mylène Reiners
• Architect @Atos
• Focus innovation

3. Sketching the context
• Big Data
• New insights
• Analytics
• Data discovery

4. Sketching the context
• Hadoop
• Storing and managing data

5. Apache Spark
• Speed
• General purpose

6. Short demo in Scala (shell)
• Simple data analysis
• Read “README.md”
• Count the number of lines

7. Role of SparkContext (sc)

8. RDD
• Resilient Distributed Dataset
• Creation
• Transformations
• Actions

9. RDD
• Lazy
• Recomputed

10. Java example (accumulator)
JavaRDD<String> rdd = sc.textFile(args[1]);
final Accumulator<Integer> blankLines = sc.accumulator(0);
JavaRDD<String> callSigns = rdd.flatMap(
new FlatMapFunction<String, String>() {
public Iterable<String> call(String line) {
if (line.equals("")) {
blankLines.add(1);
}
return Arrays.asList(line.split(" "));
}});
callSigns.saveAsTextFile("output.txt")

11. Apache Spark stack

12. Spark SQL
• Interface for working with (semi)structured data

13. Hive example
// Import Spark SQL
import org.apache.spark.sql.hive.HiveContext;
// Or if you can't have the hive dependencies
import org.apache.spark.sql.SQLContext;
// Import the JavaSchemaRDD
import org.apache.spark.sql.SchemaRDD;
import org.apache.spark.sql.Row;
(...)
JavaSparkContext ctx = new JavaSparkContext(...);
SQLContext hiveCtx = new HiveContext(ctx);

14. Hive example (cont’d)
SchemaRDD input = hiveCtx.jsonFile(inputFile);
// Register the input schema RDD
input.registerTempTable("tweets");
// Select tweets based on the retweetCount
SchemaRDD topTweets = hiveCtx.sql(
"SELECT text, retweetCount FROM tweets ORDER BY
retweetCount LIMIT 10");

15. Spark Streaming
• Acting on data as soon as it arrives
• Dstreams

16. Example
// Create a StreamingContext with a 1-second batch size from a SparkConf
JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(1));
// Create a DStream from all the input on port 7777
JavaDStream<String> lines = jssc.socketTextStream("localhost", 7777);
// Filter our DStream for lines with "error"
JavaDStream<String> errorLines = lines.filter(new Function<String, Boolean>() {
public Boolean call(String line) {
return line.contains("error");
}});
// Print out the lines with errors
errorLines.print();

17. Example
// Start our streaming context and wait for it
// to "finish"
jssc.start();
// Wait for the job to finish
jssc.awaitTermination();

18. GraphX
• Graphdatabase

19. Example
// Load the edges as a graph
val graph = GraphLoader.edgeListFile(sc,
"followers.txt")
// Run PageRank
val ranks = graph.pageRank(0.0001).vertices
// Join the ranks with the usernames
val users = sc.textFile("users.txt").map { line =>
val fields = line.split(",")
(fields(0).toLong, fields(1))
}

20. Example
val ranksByUsername =
users.join(ranks)
.map {case (id, (username, rank))
=> (username, rank)
}
// Print the result
println(ranksByUsername.collect().mkString("n"))

21. MLib
• Machine learning

22. Thank you