Your SlideShare is downloading. ×
0
×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Building LinkedIn's Learning Platform with MongoDB

1,932

Published on

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,932
On Slideshare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
17
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 1
  • 2. Jacob Dejno Web Developer Ryan Seamons Product Manager 2
  • 3. What we will not cover: • Scaling MongoDB • High load volume and QPS • Using MongoDB for analytics • MongoDB Production/Development Deployments • Big Data. 3
  • 4. We will cover: • LearnIn, LinkedIn’s Internal Learning Portal • Using Node.js and MongoDB • Mongoose ODM • Move to Elasticsearch 4
  • 5. Quick API development using MongoDB 5
  • 6. What’s your next play? 6
  • 7. 7
  • 8. 8
  • 9. 9
  • 10. 10
  • 11. What technologies should we use? 11
  • 12. How do we leverage our current knowledge? 12
  • 13. What server and database? 13
  • 14. How did we build this… quickly? 14
  • 15. Let’s start with server selection… which one? 15
  • 16. 16
  • 17. Node.js • Server-side JavaScript • Lightweight and quick to setup • NPM package support • Extensive documentation and community • Allows for data-driven JSON template rendering • Easy REST API creation using Express.js 17
  • 18. Which database? 18
  • 19. LearnIn’s Database Needs • Support 5,000+ employees, globally • Minimal data storage needs • Many more reads that writes • Able to scale with company growth • Flexibility 19
  • 20. 20
  • 21. MongoDB • Easy to setup • NoSQL with JSON structure • Advanced querying • Flexible schema, schema-less • Extensive documentation • High Single instance thresholds 21
  • 22. 22
  • 23. { “data”: “have”, “models”: “schema” } 23
  • 24. mongoose 24
  • 25. Mongoose • Object Document Mapper with easy type casting • Quick to setup, replaces Mongo JS driver • Easy document modeling and field validation API • Business logic hooks, Custom Middleware • Mongo _id reference population 25
  • 26. var mongoose = require(‘mongoose’), ! connection = mongoose.connect('mongodb://localhost:27017/test'); 26
  • 27. var mongoose = require(‘mongoose’), Schema = mongoose.Schema ! ! function BaseSchema() {
 Schema.apply(this, arguments);
 
 this.add({
 publisher: {
 type: ObjectId,
 ref: 'User'
 }
 });
 } Document Modeling & Validation var AssetSchema = new BaseModel.BaseSchema({
 title: {
 type: String,
 required: true,
 trim: true
 },
 description: {
 type: String,
 required: true,
 trim: true
 }
 },{
 collection: 'assets'
 }); 27
  • 28. this.pre('save', function (next) {
 
 if ( !this.publisher ) {
 this.publisher = this.lastUpdatedByUser;
 } 
 // Continue on to next function in queue
 next();
 }); Business Logic Hooks 28
  • 29. function queryAssets(query, callback) {
 model
 .find(query) // query is JSON object
 .populate(‘publisher’) // populate publisher from ‘users’ collection
 .exec(function (error, resultSet) {
 return callback && callback(error, resultSet);
 });
 } NoSQL Reference Population 29
  • 30. Our Schema Design with Mongoose Normalized Data Modeling Many-to-many relationships Collection per data type 30
  • 31. var AssetSchema = new BaseModel.BaseSchema({
 title: {
 type: String,
 required: true,
 trim: true
 },
 type: {
 type: String,
 default: 'general',
 enum: typeEnums
 },
 relatedAssets: [
 {
 type: ObjectId,
 ref: 'Asset'
 }
 ]
 }, {
 collection: 'assets'
 }); 31
  • 32. CourseSchema = new BaseModel.BaseSchema({
 groups: [
 {
 title: {
 type: String
 },
 items: [
 {
 asset: {
 type: ObjectId,
 ref: 'Asset'
 },
 task: {
 type: ObjectId,
 ref: 'Task'
 }
 }
 ]
 }
 ]
 }, {
 collection: 'courses'
 }); // nested population function getCourseById(id, callback) {
 Model .findById(id) .populate('groups.items.asset groups.items.task’) .exec(callback);
 } 32
  • 33. function getAsset(req, res) {
 if (!req.params.id) {
 return res.send(500, 'Please add a asset ID.');
 }
 
 assetModel.model.findOne({
 _id: req.params.id
 }).populate('relatedAssets').exec(function (err, asset) {
 if(err){
 res.send(500, err);
 return;
 }
 res.json(200, asset);
 });
 } app.get('/api/asset/:id', controller.getAsset);Route: Controller: 33
  • 34. 34
  • 35. 35
  • 36. { "_id" : ObjectId("51c0b755d292ad5a7a000039"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "We'll discuss what is required to work in a global context at LinkedIn and how managers can become effective global leaders? ", "image_url" : "/media/images/comingsoon/ thumbnails_coming-soon3.jpg", "longdescription" : "What is required to work in a global context at LinkedIn? What are the personal capabilities that enable a global manager to succeed? How do managers begin to develop a global perspective that will enable them to become effective global leaders? Please come prepared to discuss the following questions:nn(a) In your job, what will you be doing differently to reflect the global nature of our business?n(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend we do to change that?", "state" : "draft", "tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Think Global with Arvind Rajan", "type" : "slideshow" } { "_id" : ObjectId("51c0b755d292ad5a7a00003a"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "We'll discuss how to handle difficult conversations with employees regarding performance or not meeting expectations", "image_url" : "/media/images/comingsoon/ thumbnails_coming-soon3.jpg", "longdescription" : "We'll discuss how to handle difficult conversations with employees regarding performance or not meeting expectations", "state" : "draft", "tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Handling Difficult Conversations with Cliff Rosenberg", "type" : "slideshow" } { "_id" : ObjectId("51c0b755d292ad5a7a00003b"), "academy" : "Leadership", "adder" : "", "applyContent" : "", "author" : "", "clicks" : 8, "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "Containing vital performance and career lessons for managers at every level; this book presents the remarkable findings of Gallup's massive in-depth study of great managers across a wide variety of situations.", "enableEnrichedAsset" : false, "image_url" : "/media/images/learningassets/leadership/ 12.jpg", "longdescription" : "(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend we do to change that?", "practiceContent" : "", "state" : "live", "tag_suggest" : { "input" : [ "Editor", "Leadership & Management", "Books & Articles", "Transitioning to Management", "Management & Leadership" ] }, "tags" : [ "Editor", "Leadership & Management", "Books & Articles", "Transitioning to Management", "Management & Leadership" ], "thinkContent" : "", "title" : "First, Break All the Rules: What the World's Greatest Managers Do Differently", "type" : "book", "url" : "https://linkedin.okta.com/app/template_saml_2_0/k2bnhvkoMCFSHKCKIJUA/sso/saml? RelayState=https%3A%2F%2Flinkedin.skillport.com%2Fskillportfe%2Fcustom%2Flogin%2Flinkedin%2Flogin.action%3Fcourseaction%3DLaunch%26assetid%3D_ss_book%3A15501" } { "_id" : ObjectId("51c0b755d292ad5a7a00003c"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "We'll discuss the FCS model for managing hyper-growth. Fewer things done better. Communicating the right information to the right person at the right time.", "image_url" : "/ media/images/comingsoon/thumbnails_coming-soon3.jpg", "longdescription" : "We'll discuss the FCS model for managing hyper-growth. Fewer things done better. Communicating the right information to the right person at the right time.", "state" : "draft", "tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Managing Hyper- growth with Jeff Weiner", "type" : "slideshow" } { "_id" : ObjectId("51c0b755d292ad5a7a00003d"), "academy" : "Leadership", "adder" : "", "applyContent" : "", "author" : "", "clicks" : 24, "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "David shares learnings from the leaders that have inspired him and examples from his own career. ", "enableEnrichedAsset" : false, "image_url" : "/media/images/learningassets/leadership/43.jpg", "longdescription" : "David shares learnings from the leaders that have inspired him and examples from his own career. ", "practiceContent" : "", "recommendations" : [ ObjectId("521e6fd66f13a29715000044") ], "state" : "live", "tag_suggest" : { "input" : [ "Editor", "Leadership & Management", "Videos", "Leadership 101", "Leaders Teaching Leaders", "LTL", "Management & Leadership" ] }, "tags" : [ "Editor", "Leadership & Management", "Videos", "Leadership 101", "Leaders Teaching Leaders", "LTL", "Management & Leadership" ], "thinkContent" : "", "title" : "Leadership 101 with David Henke", "type" : "video", "url" : "http://innertube.linkedin.biz/video/Leadership101_2013_0118/index.html" } { "_id" : ObjectId("51c0b755d292ad5a7a00003e"), "academy" : "Professional", "author" : "EMEA", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "Learn how to apply tips and tools for using your time more efficiently, meeting deadlines and increasing productivity, including learning how to use MS Outlook more effectively.nn", "image_url" : "/media/images/learningassets/professional/36_time_management.jpg", "longdescription" : "long description", "options" : [ "hidden" ], "state" : "draft", "tag_suggest" : { "input" : [ "Professional Development", "Learning Programs & Classes" ] }, "tags" : [ "Professional Development", "Learning Programs & Classes" ], "title" : "Time Management ", "type" : "article", "url" : "https://linkedin.okta.com/app/template_saml_2_0/k166h5vsMZMIUWGSDZSJ/sso/saml?RelayState=%2fdeeplink %2fssodeeplink.aspx%3fmodule%3dtranscript%26loid%3d919c0d74-4d51-43bd-87c9-546896287d49" } { "_id" : ObjectId("51c0b755d292ad5a7a000039"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "We'll discuss what is required to work in a global context at LinkedIn and how managers can become effective global leaders? ", "image_url" : "/media/images/comingsoon/ thumbnails_coming-soon3.jpg", "longdescription" : "What is required to work in a global context at LinkedIn? What are the personal capabilities that enable a global manager to succeed? How do managers begin to develop a global perspective that will enable them to become effective global leaders? Please come prepared to discuss the following questions:nn(a) In your job, what will you be doing differently to reflect the global nature of our business?n(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend How do we search? 36
  • 37. > db.collection.find({ title: new RegExp(‘<search query>’, ‘i’) }); 37
  • 38. Full-Text Search 38
  • 39. 39 > db.collection.ensureIndex({ <field>: “text" })
  • 40. 40 > db.collection.find({ $text: { $search: <string> } });
  • 41. • Easy searching by creating MongoDB index • Relevancy scoring • Stemming and multi-language support What text search gives us: 41
  • 42. • Single tokenizer/analyzer • Simple relevancy scoring, but not using Lucene • No completion suggestion • No fuzzy matching • No related item search Some limitations… 42
  • 43. What we needed: • Lucene index relevancy scoring and performance • Custom field analyzers for tokenization and stemming • ‘Related to’ or ‘More like this’ querying • Quick completion suggestions • Complicated wildcard searching • Easy Node.js integration 43
  • 44. 44
  • 45. 45 ?
  • 46. elasticsearch-river-mongodb 46
  • 47. How does it work? 47
  • 48. Replica Set OpLog Primary Secondary Replica Index ES River 48
  • 49. Configuring the River $ curl -XPUT “localhost:9200/_river/<RIVER_NAME>/_meta” -d ‘ {
 "type": "mongodb",
 "mongodb": {
 "servers": [
 {
 "host": "< HOST >",
 "port": "< PORT >"
 }
 ],
 "db": "learnin",
 "options": {
 "import_all_collections": "true"
 }
 },
 "index": {
 "name": "prodindexnew"
 }
 }’ 49
  • 50. http://localhost:9200/index/type/_search?q=<search> 50
  • 51. elasticsearch.js 51
  • 52. Some limitations… 52
  • 53. 53
  • 54. mongoose LearnIn API 54
  • 55. 55
  • 56. Contact linkedin.com/in/rseamons ! @ryanseamons linkedin.com/in/dejno ! @dejno Jacob Dejno Web Developer Ryan Seamons Product Manager 56
  • 57. Questions? 57
  • 58. Appendix • MongoDB NoSQL Database • Mongoose ODM • Node.js server • Express.js WebApp Framework • MongoDB Full-Text Search • Elasticsearch • Elasticsearch-River-MongoDB • Elasticsearch Rivers • These Slides 58
  • 59. ©2014 LinkedIn Corporation. All Rights Reserved. 59

×