How MongoDB Integrates with Spark

1. MongoDB + Spark
@blimpyacht

2. Level Setting

6. TROUGH OF
DISILLUSIONMENT

7. Interactive Shell
Easy (-er)
Caching

8. HDFS
Distributed Data

9. HDFS
YARN
Hive
Pig
Domain Specific Languages
MapReduce

10. Spark
Stand
Alone
YARN
Mesos
HDFS
Distributed Resources

11. YARN
Spark
Mesos
HDFS
Spark
Stand
Alone
Hadoop
Distributed Processing

12. YARN
Spark
Mesos
Hive
Pig
HDFS
Hadoop
Spark
Stand
Alone
Domain Specific Languages

13. YARN
Spark
Mesos
Hive
Pig
Spark
SQL
Spark Shell
Spark
Streaming
HDFS
Spark
Stand
Alone
Hadoop

14. YARN
Spark
Mesos
Hive
Pig
Spark
SQL
Spark Shell
Spark
Streaming
HDFS
Spark
Stand
Alone
Hadoop

15. YARN
Spark
Mesos
Hive
Pig
Spark
SQL
Spark Shell
Spark
Streaming
HDFS
Spark
Stand
Alone
Hadoop

16. YARN
Spark
Mesos
Hive
Pig
Spark
SQL
Spark Shell
Spark
Streaming
Spark
Stand
Alone
Hadoop

17. Stand
Alone
YARN
Spark
Mesos
Hive
Pig
Spark
SQL
Spark
Shell
Spark
Streaming
MapReduce

18. Stand
Alone
YARN
Spark
Mesos
Spark
SQL
Spark
Shell
Spark
Streaming

19. Stand
Alone
YARN
Spark
Mesos
Spark
SQL
Spark
Shell
Spark
Streaming

20. executor
Worker Node
executor
Worker Node
Master
Java Driver
Hadoop
Connector
Driver Application

21. Parallelization
Parellelize = x

22. Transformations
Parellelize = x t(x) = x’ t(x’) = x’’

23. Transformations
filter( func )
union( func )
intersection( set )
distinct( n )
map( function )

24. Action
f(x’’) = yParellelize = x t(x) = x’ t(x’) = x’’

25. Actions
collect()
count()
first()
take( n )
reduce( function )

26. Lineage
f(x’’) = yParellelize = x t(x) = x’ t(x’) = x’’

27. Transform Transform ActionParallelize
Lineage

28. Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Lineage

29. Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Lineage

30. Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Transform Transform ActionParallelize
Lineage
http://www.blimpyacht.
com/2016/02/03/a-visual-guide-to-
the-spark-hadoop-ecosystem/

31. https://github.com/mongodb/mongo-hadoop

32. Spark Configuration
Configuration conf = new Configuration();
conf.set(
"mongo.job.input.format",
"com.mongodb.hadoop.MongoInputFormat”
);
conf.set(
"mongo.input.uri",
"mongodb://localhost:27017/db.collection”
);

33. Spark Context
JavaPairRDD<Object, BSONObject> documents = context.
newAPIHadoopRDD(
conf,
MongoInputFormat.class,
Object.class,
BSONObject.class
);

34. Spark Submit
/usr/local/spark-1.5.1/bin/spark-submit
--class com.mongodb.spark.examples.DataframeExample
--master local Examples-1.0-SNAPSHOT.jar

35. Stand
Alone
YAR
N
Spark
Meso
s
Spark
SQL
Spark
Shell
Spark
Streaming

36. JavaRDD<Message> messages = documents.map (
new Function<Tuple2<Object, BSONObject>, Message>() {
public Message call(Tuple2<Object, BSONObject> tuple) {
BSONObject header =
(BSONObject)tuple._2.get("headers");
Message m = new Message();
m.setTo( (String) header.get("To") );
m.setX_From( (String) header.get("From") );
m.setMessage_ID( (String) header.get( "Message-ID" )
);
m.setBody( (String) tuple._2.get( "body" ) );
return m;
}
}
);

39. THE FUTURE
AND
BEYOND THE INFINITE

40. Spark Connector

41. Aggregation Filters
$match | $project | $group

42. Data Locality mongos

43. THANKS!
@blimpyacht