The document discusses Apache Spark and its integration with MongoDB. It provides an overview of Spark's architecture and capabilities including Spark SQL, streaming, machine learning libraries. It then covers use cases and benefits of using Spark with MongoDB, including real-time analytics, fraud detection, and time series analysis. The document demonstrates how the Stratio Spark-MongoDB connector allows querying and analyzing MongoDB data using Spark SQL and DataFrames.

2. Spark in the Leaf

5. "BigDataSpain (2014) is great, hope I can make it
next year too"
Wish making conference!

6. "BigDataSpain (2015) is great, hope I can win la
Loteria"

7. 7
Agenda
Spark + MongoDB
Connectors
Use Cases
Demo

8. By now, you should have heard about MongoDB
Unless you've been living under a rock for the last few years!

9. 9
MongoDB
GENERAL PURPOSE DOCUMENT DATABASE OPEN-SOURCE

10. MongoDB is Fully Featured

11. 11
Apache Spark

12. 12
Spark is Taylor Swift of Big Data

13. 13
Agenda
Spark Taylor Swift + MongoDB
Connectors
Use Cases
Demo

14. 14
Apache Spark Taylor Swift

15. Spark Stack
Spark SQL
Spark
Streaming
MLIB GraphX
Apache Spark
Seamless integration
with SQL using
DataFrame API. Also
supports HIVE SQL
Fast Feed data processing API.
Designed for Fault Tolerance and
bridges streaming with batch processing
MLib is Spark machine
learning algorithms trick bag.
Spark graph library

16. Spark Stack
Spark
SQL
Spark
Streaming
MLIB GraphX
Apache Spark

17. Spark Stack
Spark
SQL
Spark
Streaming
MLIB GraphX
Apache Spark

18. Spark Stack
Spark
SQL
Spark
Streaming
MLIB GraphX
Apache Spark

19. Spark + MongoDB

20. 20
Data Management
Offline Processing
Analytics
Data Warehousing
OLTP
Applications
Fine grained operations

21. 21
Delivering User Relevancy
• Integrate data from many
sources
• Fast-cycle analytics
• Real-time
• Reliable

22. Fraud Detection
I'm so in love!

23. Fraud Detection
I'm so in love!
Me, too<3
Now send me your CC
number
?
Ok, XXXX-123-zzz
$$$

24. Fraud Detection

25. Workloads
Chat App
Login
User Profile
Contacts
Messages
…
Spark
Fraud Detection
Segmentation
Recommendations
HDFS HDFS HDFS Archiving
Data Crunching

26. 26
Wearable Devices
Embedded Systems
Internet of Things
Embedded medical devices

27. 27
Access complete patient history
Avoid of conflicting prescriptions
Clinical trials

28. High Speed Document Design

29. Time Series
db.ticks.find()
{
_id: 'MSFT_12',
type: 'Open',
date: ISODate("2015-07-12 10:00"),
volume: 1699342,
minutes: {
"0": 12.9,
"1": 14.4,
...
"59": 15.8
}
}
Resource
Type
When
Series

30. http://cdn.theatlantic.com/static/infocus/ngt051713/n10_00203194.jpg
WiredTiger

31. Very High Speed

32. > mongod --storageEngine wiredTiger

33. > mongod
On Upcoming 3.2

34. 34
MongoDB Storage Engines
Content
Repo
IoT Sensor
Backend
Ad Service
Customer
Analytics
Archive
MongoDB Query Language (MQL) + Native Drivers
MongoDB Document Data Model
MMAP V1 WT In-Memory ? ?
Supported in MongoDB 3.0 Future Possible Storage Engines
Management
Security
Experimental

35. Spark Streaming

36. 36
Spark Streaming
SparkTwitter Feed

37. 37
Spark Streaming
Twitter Feed
{
"statuses": [
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Mon Sep 24 03:35:21 +0000
2012",
"id_str": "250075927172759552",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": []
}
}
}

38. 38
Spark Streaming
Spark
{
"statuses": [
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Mon Sep 24 03:35:21 +0000
2012",
"id_str": "250075927172759552",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": []
}
}
}
{
"time": "Mon Sep 24 03:35",
"freebandnames": 1
}
{
"statuses": [
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Mon Sep 24 03:35:21 +0000
2012",
"id_str": "250075927172759552",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": []
}
}
}
{
"statuses": [
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Mon Sep 24 03:35:21 +0000
2012",
"id_str": "250075927172759552",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": []
}
}
}
{
"statuses": [
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Mon Sep 24 03:35:21 +0000
2012",
"id_str": "250075927172759552",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": []
}
}
}
{
"time": "Mon Sep 24 03:35",
"freebandnames": 4
}

39. 39
Capped Collection
Spark Streaming
{
"time": "Mon Sep 24 03:35",
"freebandnames": 4
}
{
"time": "Mon Sep 24 03:40",
"bigdataspain": 400
}
{
"time": "Mon Sep 24 03:50",
"bigdataspain": 7556
}
{
"time": "Mon Sep 24 03:50",
"itshappending": 100
}
Tailable Cursor

40. Spark SQL

41. MongoDB Hadoop Connector
Spark
HDFS HDFS HDFS
MongoDB Hadoop
Connector
MongoDB
Shard

42. MongoDB Hadoop Connector
Spark
HDFS HDFS HDFS
MongoDB Hadoop
Connector
MongoDB
Shard
YARN

43. 43
MongoDB Hadoop Connector
Positive Not So Good
Battle Tested Not the fastest thing
Integrated with existing
Hadoop components
Not dedicated to Spark
Supports HIVE and PIG Dependent on HDFS
http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-hadoop/

44. 44
Stratio Spark-MongoDB
http://spark-packages.org/?q=mongodb

45. 45
Stratio Spark-MongoDB
https://github.com/Stratio/spark-mongodb
Spark
HDFS HDFS HDFS
MongoDB
Shard
Stratio Spark-MongoDB

46. 46
Stratio Spark-MongoDB
val mcInputBuilder = MongodbConfigBuilder(Map(Host -> List("localhost:27017"),
Database -> "marketdata",
Collection -> "minbars",
SamplingRatio -> 1.0,
WriteConcern -> MongodbWriteConcern.Normal))
val readConfig = mcInputBuilder.build()
Database
Collection
Sampling Ratio
Write Concern

47. 47
Stratio Spark-MongoDB
val sqlContext = new HiveContext(sc)
val dfOneMin = sqlContext.fromMongoDB(readConfig)

48. 48
Stratio Spark-MongoDB
val dfFiveMinForMonth = sqlContext.sql(
"""
SELECT m.Symbol, m.OpenTime as Timestamp, m.Open, m.High, m.Low, m.Close
FROM
...
FROM minbars)
as m
WHERE unix_timestamp(m.CloseTime, 'yyyy-MM-dd HH:mm') - unix_timestamp(m.OpenTime,
'yyyy-MM-dd HH:mm') = 60*4"""
)

49. 49
Stratio Spark-MongoDB
https://github.com/Stratio/spark-mongodb
Spark
HDFS HDFS HDFS
MongoDB
Shard
Stratio Spark-MongoDB

50. 50
DC West
DC West
DC West
Stratio Spark-MongoDB
https://github.com/Stratio/spark-mongodb
Spark
MongoDB
Shard
Spark
Spark

51. Demo

52. 52
Demo
Spark Stratio Spark-MongoDB

53. 53
Feeling powerful ???

54. Future

55. 55
What to expect
• We are working on a dedicated Spark Connector for
MongoDB
• Stratio Connector is great but:
–Some Operations are actually faster if performed using
Aggregation Framework
• Better Integration with upcoming 3.2 Async Java Driver
–Specially for the Apache Streaming Support

56. MongoDB Days 2015
05 November, 2015 London
https://www.mongodb.com/events/mongodb-days-uk

57. 57
Engineering
Sales & Account Management Finance & People Operations
Pre-Sales Engineering Marketing
Join the Team
View all jobs and apply: http://grnh.se/pj10su

58. Obrigado!
Norberto Leite
Technical Evangelist
norberto@mongodb.com
@nleite