MongoDB and Hadoop work powerfully together as complementary technologies. Learn how the Hadoop connector allows you to leverage the power of MapReduce to process data sourced from your MongoDB cluster.

1. What’s the Scoop with Hadoop?
How the connector works and how to work it
{ Name: ‘Bryan Reinero’,
Title: ‘Developer Advocate’,
Twitter: ‘@blimpyacht’,
Email: ‘bryan@mongdb.com’ }

2. 2

3. 3
Hadoop
A framework for distributed processing of large data sets
• Terabyte and petabyte datasets
• Data warehousing
• Advanced analytics
• Not a database
• No indexes
• Batch processing

4. 4
Data Management

5. 5
Data Management
Hadoop
Fault tolerance
Batch processing
Coarse-grained operations
Unstructured Data
MongoDB
High availability
Mutable data
Fine-grained operations
Flexible Schemas

6. 6
Data Management
Hadoop
Offline Processing
Analytics
Data Warehousing
MongoDB
Online Operations
Application
Operational

7. 7
Typical Implementations
Application Server

8. 8
MongoDB as an Operational Store
Application Server

9. 9
Use Cases
• Behavioral analytics
• Segmentation
• Fraud detection
• Prediction
• Pricing analytics
• Sales analytics

10. What does it do?

11. 11
Processing Sensor Data
{
"_id" : ObjectId("556172a53004b760dde8a488"),
”deviceId" : 556172530004,
"value" : 6205,
"timestamp" : ISODate(”2015-06-02T02:03:17.906Z"),
"loc" : [
-174.95596353219008,
40.654427078258834
]
} Average Sensor Value By
Device
Time Interval
Location Bucket

12. 12
Processing Sensor Data
{
"_id" : ObjectId("556172a53004b760dde8a488"),
”deviceId" : 556172530004,
"value" : 6205,
"timestamp" : ISODate(”2015-06-02T02:03:17.906Z"),
"loc" : [
-174.95596353219008,
40.654427078258834
]
} Average Sensor Value By
Device
Time Interval
Location Bucket

13. 13
Processing Sensor Data
{
"_id" : ObjectId("556172a53004b760dde8a488"),
”deviceId" : 556172530004,
"value" : 6205,
"timestamp" : ISODate(”2015-06-02T02:03:17.906Z"),
"loc" : [
-174.95596353219008,
40.654427078258834
]
} Average Sensor Value By
Device
Time Interval
Location Bucket

14. 14
Processing Sensor Data
{
"_id" : ObjectId("556172a53004b760dde8a488"),
”deviceId" : 556172530004,
"value" : 6205,
"timestamp" : ISODate(”2015-06-02T02:03:17.906Z"),
"loc" : [
-174.95596353219008,
40.654427078258834
]
} Average Sensor Value By
Device
Time Interval
Location Bucket

15. 15
Processing Sensor Data
{
"_id" : ObjectId("556172a53004b760dde8a488"),
"d_id" : ObjectId("556172a53004b760dde8a443"),
"v" : 6205,
"timestamp" : ISODate("3129-12-13T02:03:17.906Z"),
"loc" : [
-174.95596353219008,
40.654427078258834
]
}
LIVE CODE DEMO!!!
PSUEDO
^

16. 16
MapReduce
map() {
emit(
{ key: ObjectId(…),
value: 6205
}
);
emit(
{ key: Chelsea,
value: 6205
}
);
emit(
{ key: m06_d01_h02,
value: value
}
);
}

17. 17
MapReduce
{key: ObjectId(…),
value: 6205 }
map() {
emit(
{ key: ObjectId(…),
value: 6205
}
);
emit(
{ key: Chelsea,
value: 6205
}
);
emit(
{ key: m06_d01_h02,
value: value
}
);
}

18. 18
MapReduce
map() {
emit(
{ key: ObjectId(…),
value: 6205
}
);
emit(
{ key: Chelsea,
value: 6205
}
);
emit(
{ key: m06_d01_h02,
value: value
}
);
}
{key: Chelsea,
value: 6205}

19. 19
MapReduce
map() {
emit(
{ key: ObjectId(…),
value: 6205
}
);
emit(
{ key: Chelsea,
value: 6205
}
);
emit(
{ key: m06_d01_h02,
value: value
}
);
}
{ key: m06_d01_h02,
value: 6205}

20. 20
MapReduce

21. 21
MapReduce
key: Chelsea, value: 6025

22. 22
MapReduce
key: Chelsea, value: 4904

23. 23
MapReduce
key: Chelsea, value: 6338

24. 24
MapReduce
key: m06_d01_h02, value: 6205

25. 25
MapReduce
key: m06_d01_h02, value: 4904

26. 26
MapReduce
key: m06_d01_h02, value: 6338

27. 27
MapReduce
key: m06_d01_h02, value: 6721

28. 28
MapReduce
function reduce ( key, values ) {
var result = { count: 1, sum : 0 };
values.forEach( function( v ){
result.sum = v.value;
result.count++;
});
return result;
}

29. 29
MapReduce
function reduce ( key, values ) {
var result = { count: 1, sum : 0 };
values.forEach( function( v ){
result.sum = v.value;
result.count++;
});
return result;
}

30. 30
HDFS
YARN
MapReduce
Pig Hive
Spark

31. 31
HDFS and YARN
• Hadoop Distributed File System (HDFS)
– Distributed file-system that stores data on commodity machines
in a Hadoop cluster
• Yet Another Resource Negotiator (YARN)
– Resource management platform responsible for managing and
scheduling compute resources in a Hadoop cluster

32. 39
Hadoop Distributed File System (HDFS)
DATA
NODE
DATA
NODE
DATA
NODE
DATA
NODE
Client
Read / Writes
Replication
NAME
NODE
Metadata
Operations

33. 47
Yet Another Resource Negotiator
COMPUTE NODE
Client
NODE
MANAGER
NODE
MANAGER
RESOURCE
MANAGER
APPLICATION
MASTER
CONTAINER
CONTAINER
CONTAINER

34. Using The Connector

35. 49
What You’re Gonna Need
A reducer class
extends org.apache.hadoop.mapreduce.Reducer
A mapper class
extends org.apache.hadoop.mapreduce.Mapper

36. 50
MapReduce Configuration
• MongoDB input
– mongo.job.input.format = com.mongodb.hadoop.MongoInputFormat
– mongo.input.uri = mongodb://mydb:27017/db1.collection1
• MongoDB output
– mongo.job.output.format = com.mongodb.hadoop.MongoOutputFormat
– mongo.output.uri = mongodb://mydb:27017/db1.collection2
• BSON input/output
– mongo.job.input.format = com.hadoop.BSONFileInputFormat
– mapred.input.dir = hdfs:///tmp/database.bson
– mongo.job.output.format = com.hadoop.BSONFileOutputFormat
– mapred.output.dir = hdfs:///tmp/output.bson

37. 51
Yet Another Resource Negotiator
COMPUTE NODE
Client
NODE
MANAGER
NODE
MANAGER
RESOURCE
MANAGER
APPLICATION
MASTER
CONTAINER
CONTAINER
CONTAINER
Bin/hadoop jar MyJob.jar
MongoDB_Hadoop_Connector.jar

38. 52

39. 53
extends MongoSplitter class

40. 54
extends MongoSplitter class
List<InputSplit> calculateSplits()

41. 55
Cluster
MONGOS
SHARD A
SHARDB
SHARD C
SHARD D
MONGOS Client

42. 56
• High-level platform for creating MapReduce
• Pig Latin abstracts Java into easier-to-use notation
• Executed as a series of MapReduce applications
• Supports user-defined functions (UDFs)
Pig

43. 57
samples = LOAD 'mongodb://127.0.0.1:27017/sensor.logs'
USING
com.mongodb.hadoop.pig.MongoLoader(’deviceId:int,value:double');
grouped = GROUP samples by deviceId;
sample_stats = FOREACH grouped {
mean = AVG(samples.value);
GENERATE group as deviceId, mean as mean;
}
STORE sample_stats INTO 'mongodb://127.0.0.1:27017/sensor.stats'
USING com.mongodb.hadoop.pig.MongoStorage;

44. 58
• Data warehouse infrastructure built on top of Hadoop
• Provides data summarization, query, and analysis
• HiveQL is a subset of SQL
• Support for user-defined functions (UDFs)

45. 59
• Powerful built-in transformations and actions
– map, reduceByKey, union, distinct, sample, intersection, and more
– foreach, count, collect, take, and many more
An engine for processing Hadoop data. Can perform
MapReduce in addition to streaming, interactive queries,
and machine learning.

46. 60
Data Flows
Hadoop
Connector
BSON Files
MapReduce & HDFS

47. Thanks!
{ name: ‘Bryan Reinero’,
title: ‘Developer Advocate’,
twitter: ‘@blimpyacht’,
code: ‘github.com/breinero’
email: ‘bryan@mongdb.com’ }