Download as PDF, PPTX
![Documents Support Modern Requirements
Relational Document Data Structure
{ customer_id : 123,
first_name : "Mark",
last_name : "Smith",
city : "San Francisco",
location : [37.7576792,-122.5078112],
image : <binary data>,
phones: [ {
number : “1-212-555-1212”,
dnc : true,
type : “home”
},
{
number : “1-212-555-1213”,
type : “cell”
} ]
}](https://image.slidesharecdn.com/mongodbandhadoopdallas20160330-160331223615/75/MongoDB-Evenings-Dallas-What-s-the-Scoop-on-MongoDB-Hadoop-5-2048.jpg)
![db.tweets.aggregate([
{$group: {
_id: {
hour: {$hour: "$date"},
minute: {$minute: "$date"}
},
total: {$sum: "$sentiment.score"},
average: {$avg: "$sentiment.score"},
count: {$sum: 1},
happyTalk: {$push: "$sentiment.positive"}
}},
{$unwind: "$happyTalk"},
{$unwind: "$happyTalk"},
{$group: {
_id: "$_id",
total: {$first: "$total"},
average: {$first: "$average"},
count: {$first: "$count"},
happyTalk: {$addToSet: "$happyTalk"}
}},
{$sort: {_id: -1} }
])
But doesn't
MongoDB have…?
• aggregation framework
– machine learning libraries
• map reduce
– Javascript
– competing workloads](https://image.slidesharecdn.com/mongodbandhadoopdallas20160330-160331223615/75/MongoDB-Evenings-Dallas-What-s-the-Scoop-on-MongoDB-Hadoop-18-2048.jpg)
![Create the Resilient Distributed Dataset (RDD)
rdd = sc.newAPIHadoopRDD(
config, MongoInputFormat.class, Object.class, BSONObject.class)
config.set(
"mongo.input.uri", "mongodb://127.0.0.1:27017/marketdata.minbars")
config.set(
"mongo.input.query", '{"_id":{"$gt":{"$date":1182470400000}}}')
config.set(
"mongo.output.uri", "mongodb://127.0.0.1:27017/marketdata.fiveminutebars")
val minBarRawRDD = sc.newAPIHadoopRDD(
config,
classOf[com.mongodb.hadoop.MongoInputFormat],
classOf[Object],
classOf[BSONObject])](https://image.slidesharecdn.com/mongodbandhadoopdallas20160330-160331223615/75/MongoDB-Evenings-Dallas-What-s-the-Scoop-on-MongoDB-Hadoop-39-2048.jpg)
Uploaded byMongoDB
MongoDB Evenings Dallas: What's the Scoop on MongoDB & Hadoop
The document outlines the agenda for an event titled 'MongoDB Evenings Dallas,' which includes networking, discussions on MongoDB and Hadoop, and insights from industry speakers. It highlights MongoDB's technical capabilities, its integration with Hadoop for data processing, and various use cases in commerce and insurance. Additionally, it provides details on MongoDB connectors, data operations, analytics landscape, and event registration information.
More Related Content
Similar to MongoDB Evenings Dallas: What's the Scoop on MongoDB & Hadoop
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DevFest Strasbourg 2025] - NodeJs Can do that !!](https://cdn.slidesharecdn.com/ss_thumbnails/devfeststrasbourg2025-nodejscandothat-251127142731-da65b6fd-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Artificial Intelligence] AI for the Underdogs: Innovation for Sma...](https://cdn.slidesharecdn.com/ss_thumbnails/ai-aifortheunderdogsinnovationforsmallbusinesses-251124030839-72a599a4-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] Digital Accessibility: Why Developers nee...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-digitalaccessibilitywhydevelopersneedtoknowandcarein2025-251127011019-0674441d-thumbnail.jpg?width=640&height=640&fit=bounds)
![[BDD 2025 - Full-Stack Development] PHP in AI Age: The Laravel Way. (Rizqy Hi...](https://cdn.slidesharecdn.com/ss_thumbnails/fs-phpinaiagethelaravelway-251125012602-ef9d330e-thumbnail.jpg?width=640&height=640&fit=bounds)
MongoDB Evenings Dallas: What's the Scoop on MongoDB & Hadoop
- 1. Tweet tonight using#MongoDB to be entered to win a t-shirt!
- 2. Welcome to MongoDBEvenings Dallas! Agenda 5:30pm: Pizza, Beer & Soft Drinks 6:00pm: Welcome William Kent, Enterprise Account Executive, MongoDB 6:10pm: What’s the Scoop on MongoDB & Hadoop? Jake Angerman, Senior Solutions Architect, MongoDB 7:00pm:Acxiom's Journey with MongoDB John Riewerts, Director of Engineering - Marketing Services, Acxiom 7:45pm:Announcements Q&A
- 3. MongoDB + Hadoop {Name: ‘Jake Angerman’, Title: ‘Sr. Solutions Architect’}
- 4.
- 5. Documents Support ModernRequirements Relational Document Data Structure { customer_id : 123, first_name : "Mark", last_name : "Smith", city : "San Francisco", location : [37.7576792,-122.5078112], image : <binary data>, phones: [ { number : “1-212-555-1212”, dnc : true, type : “home” }, { number : “1-212-555-1213”, type : “cell” } ] }
- 6. 6 MongoDB Technical Capabilities Applica'on Driver Mongos Primary Secondary Secondary Shard 1 Primary Secondary Secondary Shard 2 … Primary Secondary Secondary Shard N 1. Dynamic Document Schema { name: “John Smith”, date: “2013-08-01”, address: “10 3rd St.”, phone: { home: 1234567890, mobile: 1234568138 } } db.customer.insert({…}) db.customer.find({ name: ”John Smith”}) 2. Na3ve language drivers 4. High performance - Data locality - Indexes - RAM 3. High availability - Replica sets 5. Horizontal scalability - Sharding
- 7. 7 Hadoop A framework fordistributed processing of large data sets • Terabyte and petabyte datasets • Data warehousing • Advanced analytics • Not a database • No indexes • Batch processing
- 8.
- 9. 9 Data Management Hadoop Fault tolerance Batchprocessing Coarse-grained operations Unstructured Data WORM MongoDB High availability Mutable data Fine-grained operations Flexible Schemas CRUD
- 10. 10 Data Management Hadoop Offline Processing Analytics DataWarehousing MongoDB Online Operations Application Operational
- 11. 11 Commerce Use Case Applications poweredby Analysis powered by • Products & Inventory • Recommended products • Customer profile • Session management • Elastic pricing • Recommendation models • Predictive analytics • Clickstream history MongoDB Connector for Hadoop
- 12. 12 Insurance Use Case Applications poweredby Analysis powered by • Customer profiles • Insurance policies • Session data • Call center data • Customer action analysis • Churn analysis • Churn prediction • Policy rates MongoDB Connector for Hadoop
- 13. 13 Fraud Detection UseCase Payments Fraud modeling Nightly Analysis MongoDB Connector for Hadoop Results Cache Online payments processing 3rd Party Data Sources Fraud Detection query only query only
- 14. Where MongoDB Fits inthe Hadoop Ecosystem
- 15.
- 16.
- 17. 17 MongoDB Connector forHadoop • Low latency • Rich fast querying • Flexible indexing • Aggregations in database • Known data relationships • Great for any subset of data • Longer jobs • Batch analytics • Highly parallel processing • Unknown data relationships • Great for looking at all data or large subsets Applications Distributed Analytics MongoDB Connector for Hadoop
- 18. db.tweets.aggregate([ {$group: { _id: { hour: {$hour: "$date"}, minute: {$minute: "$date"} }, total: {$sum: "$sentiment.score"}, average: {$avg: "$sentiment.score"}, count: {$sum: 1}, happyTalk: {$push: "$sentiment.positive"} }}, {$unwind: "$happyTalk"}, {$unwind: "$happyTalk"}, {$group: { _id: "$_id", total: {$first: "$total"}, average: {$first: "$average"}, count: {$first: "$count"}, happyTalk: {$addToSet: "$happyTalk"} }}, {$sort: {_id: -1} } ]) But doesn't MongoDB have…? • aggregation framework – machine learning libraries • map reduce – Javascript – competing workloads
- 19. 19 MongoDB Data OperationsSpectrum • Document Retrieval – 1ms if in cache, ~10ms from spinning disk • .find() – per-document cost similar to single document – _id range – any secondary index range, can be composite key – intersect two indexes – covered indexes even faster • .count(), .distinct(), .group() – fast, may be covered • .aggregate() – retrieval cost like find, plus pipeline operations – $match, $group – $project, $redact • .mapReduce() – in-database Javascript • Hadoop Connector – mongo.input.query for indexed partial scan – full scan Faster…………….....Slower
- 20.
- 21. 21 Analytics Landscape Batch /Predictive / Ad Hoc (mins – hours) Real-Time Dashboards / Scoring (<30 ms) Planned Reporting (secs – mins ) Modern Legacy
- 22.
- 23. MetLife – SingleView … Single CSR Applica3on Unified Customer Portal Opera3onal Repor3ng Cards …Cards Silo 1 … Opera'onal Data Layer • Insurance policies • Demographic data • Customer web data • Call center data DW/Data Lake • Churn predic3on algorithms MongoDB Connector for Hadoop Cards Cards Silo 2 Cards Cards Silo N Pub-sub/ETL Customer Clustering Churn Analysis Predic3ve analy3cs …
- 24. 25 Foursquare • k-nearest neighborproblems – similarity of venues, people, or brands • MongoDB data has advantages when used with MapReduce – log files can be stale – log files may not contain as much information – you can scan much less data BSON dump MongoDB Connector for Hadoop
- 25. MongoDB Connector forHadoop
- 26.
- 27.
- 28. 30 MapReduce Configuration • MongoDBinput – mongo.job.input.format = com.mongodb.hadoop.MongoInputFormat – mongo.input.uri = mongodb://mydb:27017/db1.collection1 • MongoDB output – mongo.job.output.format = com.mongodb.hadoop.MongoOutputFormat – mongo.output.uri = mongodb://mydb:27017/db1.collection2 • BSON input/output – mongo.job.input.format = com.hadoop.BSONFileInputFormat – mapred.input.dir = hdfs:///tmp/database.bson – mongo.job.output.format = com.hadoop.BSONFileOutputFormat – mapred.output.dir = hdfs:///tmp/output.bson
- 29. 33 MongoDB Cluster MONGOS SHARD A SHARDB SHARD C SHARD D MONGOS Client
- 30.
- 31. 35 extends MongoSplitter class List<InputSplit>calculateSplits()
- 32. 36 • High-level platformfor creating MapReduce • Pig Latin abstracts Java into easier-to-use notation • Executed as a series of MapReduce applications • Supports user-defined functions (UDFs) Pig
- 33. 37 samples = LOAD'mongodb://127.0.0.1:27017/sensor.logs' USING com.mongodb.hadoop.pig.MongoLoader(’deviceId:int,value:double'); grouped = GROUP samples by deviceId; sample_stats = FOREACH grouped { mean = AVG(samples.value); GENERATE group as deviceId, mean as mean; } STORE sample_stats INTO 'mongodb://127.0.0.1:27017/sensor.stats' USING com.mongodb.hadoop.pig.MongoStorage;
- 34. 38 • Data warehouseinfrastructure built on top of Hadoop • Provides data summarization, query, and analysis • HiveQL is a subset of SQL • Support for user-defined functions (UDFs)
- 35. 39 Hive Support CREATE TABLE mongo_users (id int, name string, age int) STORED BY "com.mongodb.hadoop.hive.MongoStorageHandler" WITH SERDEPROPERTIES("mongo.columns.mapping” = "_id,name,age”) TBLPROPERTIES("mongo.uri" = "mongodb://host:27017/test.users”) • Accesscollections as Hive tables • Use with MongoStorageHandler or BSONStorageHandler
- 36.
- 37.
- 38. 42 • Powerful built-intransformations and actions – map, reduceByKey, union, distinct, sample, intersection, and more – foreach, count, collect, take, and many more An engine for processing Hadoop data. Can perform MapReduce in addition to streaming, interactive queries, and machine learning.
- 39. Create the ResilientDistributed Dataset (RDD) rdd = sc.newAPIHadoopRDD( config, MongoInputFormat.class, Object.class, BSONObject.class) config.set( "mongo.input.uri", "mongodb://127.0.0.1:27017/marketdata.minbars") config.set( "mongo.input.query", '{"_id":{"$gt":{"$date":1182470400000}}}') config.set( "mongo.output.uri", "mongodb://127.0.0.1:27017/marketdata.fiveminutebars") val minBarRawRDD = sc.newAPIHadoopRDD( config, classOf[com.mongodb.hadoop.MongoInputFormat], classOf[Object], classOf[BSONObject])
- 40.
- 41.
- 42. >>> mongo_rdd =sc.mongoRDD('mongodb://localhost:27017/adsb.tincan') >>> parsed_rdd = mongo_rdd.map(parseData) >>> clusters = KMeans.train(parsed_rdd, 10, maxIterations=10, runs=1, initializationMode="random") >>> cluster_sizes = parsed_rdd.map(lambda e: clusters.predict(e)).countByValue() >>> cluster_sizes defaultdict(<type 'int'>, {0: 70122, 1: 350890, 2: 118596, 3: 104609, 4: 254759, 5: 175840, 6: 166789, 7: 68309, 8: 147826, 9: 495102}) K-means Clustering
- 43.
- 44.
- 45.
- 46.
- 47. Optimal location for providing operational responsetimes & slices Governance to choose where to load and process data More Complete EDMArchitecture & Data Lake …Siloed source databases External feeds (batch) Streams Stream icon from: https://en.wikipedia.org/wiki/File:Activity_Streams_icon.png Data processing pipeline Pub-sub,ETL,fileimports Stream Processing Downstream Systems … … Single CSR Applica3on Unified Digital Apps Opera3ona l Repor3ng … … … Analy3c Repor3ng Drivers & Stacks Customer Clusterin g Churn Analysis Predic3v e Analy3cs … Distributed Processing Operational Applications & Reporting Can run processing on all data or slices Data Lake
- 48. Code “JakeAngerman” gets25% off Super Early Bird Registration Ends March 25, 2016 June 28 - 29, 2016 New York, NY www.mongodbworld.com