Download as PDF, PPTX
Uploaded byMongoDB APAC
Buildingsocialanalyticstoolwithmongodb
This document discusses building a social analytics tool using MongoDB from a developer's perspective. It covers using MongoDB for its schema-less data and ability to handle fast read-write operations. Key topics include using aggregation queries to gain insights from data by chaining queries together and filtering/manipulating results at each stage. JavaScript capabilities in MongoDB allow applying business logic directly to data. Examples demonstrate removing garbage data and stopwords. Indexes, current progress, and tips/tricks learned around cloning collections and removing vs dropping are also covered, with a demo planned.
More Related Content
Similar to Buildingsocialanalyticstoolwithmongodb
Buildingsocialanalyticstoolwithmongodb
- 1.
- 2. 1. Product Overview 2. Why MongoDB for us? 3. Aggrega?on Queries to the rescue 4. How Javascript helped us? 5. Experiences with Indexes 6. In-‐progress use-‐cases 7. Tips & Tricks 8. Demo Agenda
- 3. Abhishek Tejpaul SoUware Developer @ IntelliGrape SoUware Loves Grails, Git and Linux abhishek@intelligrape.com About me
- 4. DataSiU Instagram Web Crawler1 Web Crawler… mongoDB Product Overview – Information Flow
- 5. Product Overview –Results
- 6. Product Overview –Results
- 7. Product Overview –Results
- 8. • Schema-‐less data. Typical data sources • Adding new social pla4orms in future • Needed fast read-‐write opera6ons Why MongoDB for us?
- 9. Aggregation Queries –Getting Insights • Combina6on of queries chained together • At every stage, we can filter/chain/massage data Image credit: h@ps://www.openshiC.com/blogs/an-‐overview-‐of-‐whats-‐new-‐in-‐mongodb-‐22
- 10. Our use-case (esp.for graphs) • Sen6ment Analysis • Demographic Analysis • Ar6cle Analysis • Plan • Crea?on of Intelligence tables in advance • Reality • On-‐the-‐fly analysis using Aggrega6on queries
- 11. How to goabout it? • Operates on a single collec6on • Think about data you have and insights you want • Focus on reducing data size early on • $match • $project • $sort • $limit, $skip • Example db.collec?onName.aggregate( { "$match" : { fieldName : matchingValue }, { "$project" : { oldOrNewField: fieldValue }}, { "$group" : { fieldName : oldOrNewField, "sum": {"$sum":1}}}, { "$sort" : { "sum" : -‐1 }}, { "$limit" : 20 })
- 12. Javascript Capabilities • All the programming capabili6es of Javascript language at your disposal • Taking business logic / processing to your data-‐store
- 13. Javascript – Ouruse-cases • Remove garbage data at DB level • Twijer wrong results • Filtering out STOP keywords db.IgnoreList.findOne().stopWords.forEach( func?on(data) { db.ProcessedAr?cle.update( { "isAc?ve" : true, "isIgnored" : {"$ne":true} }, { "$pull" : {"topicOfDiscussion" : {"name": data}}, "$set" : {"isIgnored" : true} }, { "mul?" : true } ) }); return true
- 14. Javascript – Caveats • Takes up read-‐write locks on the en6re database • Can be run with {‘noLock’ : true} op?on db.runCommand({ Eval: <func?on>, Args: <args>, Nolock: <true/false> }) • Can be replaced by mapreduce in most cases • Take it as one-‐off case
- 15. Indexes – Ouruse-cases • dropDups {dropDups : true} • backGround {backGround : true} • Time to Live {expireAUerSeconds : 3600} • Compound Indexing {key1 : 1, key2 : 1} != {key1 : 1}
- 16. Our current state • Faster write opera?ons • Under high data load from different sources • Faster read opera?ons • Graph rendering up-‐to 10 x quicker • Ease of scalability • Though yet to reach there
- 17. Work In Progress • Full-‐text search implementa?on • can be created only on strings or array of strings • db.collec?onName.ensureIndex( { fieldName : "text" } ) • Capped Collec?ons • Widgets for last-‐run jobs / event log tables • Very fast writes possible • db.createCollec?on("cName", { capped : true, size : 5242880, max : 5000 } ) • size argument is always required
- 18. Tips / Tricks– Things we learnt • cloneCollec6on • No more ssh/scp to remote systems • db.runCommand({cloneCollec?on: <nsCollec?on>, from: <remote>, query: {}}) • db.cloneCollec?on(from, collec?onName, query) • db.Collec-onName.copyTo • doesn’t not copy indexes
- 19. Tips / Tricks– Things we learnt • remove() vs drop() • Can’t use remove for capped collec6ons • remove keeps indexes while drop() clears them • To remove all the documents in a collec?on, use drop() • To remove beZer part of large collec?on, use javascript • preZy() find by default • DBQuery.prototype._prejyShell = true ( inside your ~/.mongorc.js)
- 20.
- 21. I am not a MongoDB expert though J
- 22.