This document contains code for a basic word count program written in Java using Apache Spark. It defines Mapper and Reducer classes to count the frequency of words in a text file. The main method sets up the job configuration and runs the job. Other sections provide links about the history of Spark and summaries of Spark surveys from 2016 and 2017 focusing on trends in machine learning, streaming, and scaling Spark applications.

2. https://github.com/apache/spark/commits/master?after=9e50a1d37a4cf0c34e20a7c1a910ceaff41535a2+1574&author=mateiz
http://blog.madhukaraphatak.com/history-of-spark/

4. package org.myorg;
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
public class WordCount {
public static class Map extends MapReduceBase implements Mapper<Lon
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector<Tex
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
output.collect(word, one);
}
}
}
public static class Reduce extends MapReduceBase implements Reducer
public void reduce(Text key, Iterator values, OutputCollector<Tex
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
JobConf conf = new JobConf(WordCount.class);
conf.setJobName("wordcount");
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(Map.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));
JobClient.runJob(conf);
}
}
wordcount

5. file.flatMap(line => line.split(" “))
.map(word => (word, 1))
.reduceByKey(_ + _)
wordcount
Logistic regression in Hadoop and Spark

10. 2016 Spark survey

11. 2016 Spark survey

12. 2016 Spark survey

13. 2016 Spark survey
https://www.slideshare.net/SparkSummit/trends-for-big-data-and-apache-spark-in-2017-by-matei-zaharia

14. 2016 Spark survey

15. 2016 Spark survey

16. 2016 Spark survey

17. 2016 Spark survey

19. Learning 52
Streaming 39
Machine 38
Deep 33
Science 30
Analyics 27
Scale 21
Developer 20
Enterprise 20
Ecosystem 20
Research 13
Applications 9
Processing 8
Pipeline 8
Platform 7

20. 2017 Spark summithttps://www.slideshare.net/SparkSummit/trends-for-big-data-and-apache-spark-in-2017-by-matei-zaharia

21. 2017 Spark summithttps://www.slideshare.net/SparkSummit/trends-for-big-data-and-apache-spark-in-2017-by-matei-zaharia

22. • http://blog.madhukaraphatak.com/history-of-spark/
• http://cdn2.hubspot.net/hubfs/438089/DataBricks_Surveys_-_Content/2016_Spark_Survey/
2016_Spark_Infographic.pdf
• http://cdn2.hubspot.net/hubfs/438089/DataBricks_Surveys_-_Content/Spark-Survey-2015-
Infographic.pdf