Slides from LinkedIn's presentation at MongoDB World 2014, description below:
LearnIn is LinkedIn’s internal learning platform packed with a huge variety of resources that will help our employees learn, develop, and grow professionally. In this talk, we will discuss how a small team of web developers built this platform’s API using MongoDB and a full JavaScript stack including Node.js. In particular, we will look at schema design and document validation using Mongoose ODM for Node.js, as well as quick document search utilizing MongoDB full-text search and our move to ElasticSearch using the MongoDB oplog.
3. We will cover:
• LearnIn, LinkedIn’s Internal Learning Portal
• Using Node.js and MongoDB
• Mongoose ODM
• Move to Elasticsearch
3
4. What we will not cover:
• Scaling MongoDB
• High load volume and QPS
• Using MongoDB for analytics
• MongoDB Production/Development Deployments
• Big Data.
4
8. • Internal Learning Platform for LinkedIn employees
• Aggregates LinkedIn and 3rd party content
• Curated by content team for specific roles at LinkedIn
8
18. Node.js
• Server-side JavaScript
• Lightweight and quick to setup
• NPM package support
• Extensive documentation and community
• Allows for data-driven JSON template rendering
• Easy REST API creation using Express.js
18
20. LearnIn’s Database Needs
• Support 5,000+ employees, globally
• Minimal data storage needs
• Many more reads that writes
• Able to scale with company growth
• Flexibility
20
26. Mongoose
• Object Document Mapper with easy type casting
• Quick to setup
• Easy document modeling and field validation API
• Business logic hooks, Custom Middleware
• Mongo _id reference population
26
27. var mongoose = require(‘mongoose’),
!
connection = mongoose.connect('mongodb://localhost:27017/test');
27
29. this.pre('save', function (next) {
if ( !this.publisher ) {
this.publisher = this.lastUpdatedByUser;
}
// Continue on to next function in queue
next();
});
Business Logic Hooks
29
30. function queryAssets(query, callback) {
model
.find(query) // query is JSON object
.populate(‘publisher’) // populate publisher from ‘users’ collection
.exec(function (error, resultSet) {
return callback && callback(error, resultSet);
});
}
NoSQL Reference Population
30
31. Our Schema Design with Mongoose
Normalized Data Modeling
Many-to-many relationships
Collection per data type
31
37. { "_id" : ObjectId("51c0b755d292ad5a7a000039"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"),
"description" : "We'll discuss what is required to work in a global context at LinkedIn and how managers can become effective global leaders? ", "image_url" : "/media/images/comingsoon/
thumbnails_coming-soon3.jpg", "longdescription" : "What is required to work in a global context at LinkedIn? What are the personal capabilities that enable a global manager to succeed? How
do managers begin to develop a global perspective that will enable them to become effective global leaders? Please come prepared to discuss the following questions:nn(a) In your job, what
will you be doing differently to reflect the global nature of our business?n(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend
we do to change that?", "state" : "draft", "tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Think Global with Arvind Rajan", "type" :
"slideshow" }
{ "_id" : ObjectId("51c0b755d292ad5a7a00003a"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"),
"description" : "We'll discuss how to handle difficult conversations with employees regarding performance or not meeting expectations", "image_url" : "/media/images/comingsoon/
thumbnails_coming-soon3.jpg", "longdescription" : "We'll discuss how to handle difficult conversations with employees regarding performance or not meeting expectations", "state" : "draft",
"tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Handling Difficult Conversations with Cliff Rosenberg", "type" : "slideshow" }
{ "_id" : ObjectId("51c0b755d292ad5a7a00003b"), "academy" : "Leadership", "adder" : "", "applyContent" : "", "author" : "", "clicks" : 8, "dateCreated" : ISODate("2013-06-18T19:39:01Z"),
"date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "Containing vital performance and career lessons for managers at every level; this book presents the remarkable
findings of Gallup's massive in-depth study of great managers across a wide variety of situations.", "enableEnrichedAsset" : false, "image_url" : "/media/images/learningassets/leadership/
12.jpg", "longdescription" : "(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend we do to change that?", "practiceContent" : "",
"state" : "live", "tag_suggest" : { "input" : [ "Editor", "Leadership & Management", "Books & Articles", "Transitioning to Management", "Management & Leadership" ] },
"tags" : [ "Editor", "Leadership & Management", "Books & Articles", "Transitioning to Management", "Management & Leadership" ], "thinkContent" : "", "title" : "First, Break All
the Rules: What the World's Greatest Managers Do Differently", "type" : "book", "url" : "https://linkedin.okta.com/app/template_saml_2_0/k2bnhvkoMCFSHKCKIJUA/sso/saml?
RelayState=https%3A%2F%2Flinkedin.skillport.com%2Fskillportfe%2Fcustom%2Flogin%2Flinkedin%2Flogin.action%3Fcourseaction%3DLaunch%26assetid%3D_ss_book%3A15501" }
{ "_id" : ObjectId("51c0b755d292ad5a7a00003c"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"),
"description" : "We'll discuss the FCS model for managing hyper-growth. Fewer things done better. Communicating the right information to the right person at the right time.", "image_url" : "/
media/images/comingsoon/thumbnails_coming-soon3.jpg", "longdescription" : "We'll discuss the FCS model for managing hyper-growth. Fewer things done better. Communicating the right
information to the right person at the right time.", "state" : "draft", "tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Managing Hyper-
growth with Jeff Weiner", "type" : "slideshow" }
{ "_id" : ObjectId("51c0b755d292ad5a7a00003d"), "academy" : "Leadership", "adder" : "", "applyContent" : "", "author" : "", "clicks" : 24, "dateCreated" : ISODate("2013-06-18T19:39:01Z"),
"date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "David shares learnings from the leaders that have inspired him and examples from his own career. ",
"enableEnrichedAsset" : false, "image_url" : "/media/images/learningassets/leadership/43.jpg", "longdescription" : "David shares learnings from the leaders that have inspired him and
examples from his own career. ", "practiceContent" : "", "recommendations" : [ ObjectId("521e6fd66f13a29715000044") ], "state" : "live", "tag_suggest" : { "input" : [ "Editor", "Leadership &
Management", "Videos", "Leadership 101", "Leaders Teaching Leaders", "LTL", "Management & Leadership" ] }, "tags" : [ "Editor", "Leadership & Management",
"Videos", "Leadership 101", "Leaders Teaching Leaders", "LTL", "Management & Leadership" ], "thinkContent" : "", "title" : "Leadership 101 with David Henke", "type" :
"video", "url" : "http://innertube.linkedin.biz/video/Leadership101_2013_0118/index.html" }
{ "_id" : ObjectId("51c0b755d292ad5a7a00003e"), "academy" : "Professional", "author" : "EMEA", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" :
ISODate("2014-02-07T16:35:00.173Z"), "description" : "Learn how to apply tips and tools for using your time more efficiently, meeting deadlines and increasing productivity, including learning
how to use MS Outlook more effectively.nn", "image_url" : "/media/images/learningassets/professional/36_time_management.jpg", "longdescription" : "long description", "options" :
[ "hidden" ], "state" : "draft", "tag_suggest" : { "input" : [ "Professional Development", "Learning Programs & Classes" ] }, "tags" : [ "Professional Development", "Learning Programs &
Classes" ], "title" : "Time Management ", "type" : "article", "url" : "https://linkedin.okta.com/app/template_saml_2_0/k166h5vsMZMIUWGSDZSJ/sso/saml?RelayState=%2fdeeplink
%2fssodeeplink.aspx%3fmodule%3dtranscript%26loid%3d919c0d74-4d51-43bd-87c9-546896287d49" }
{ "_id" : ObjectId("51c0b755d292ad5a7a000039"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"),
"description" : "We'll discuss what is required to work in a global context at LinkedIn and how managers can become effective global leaders? ", "image_url" : "/media/images/comingsoon/
thumbnails_coming-soon3.jpg", "longdescription" : "What is required to work in a global context at LinkedIn? What are the personal capabilities that enable a global manager to succeed? How
do managers begin to develop a global perspective that will enable them to become effective global leaders? Please come prepared to discuss the following questions:nn(a) In your job, what
will you be doing differently to reflect the global nature of our business?n(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend
How do we search?
37
42. • Easy searching by creating a MongoDB index
• Relevancy scoring
• Stemming and multi-language support
What text search gives us:
42
43. • Single tokenizer/analyzer
• Simple relevancy scoring, but not using Lucene
• No completion suggestion
• No fuzzy matching
• No related item search
Some limitations…
43
44. What we needed:
• Lucene index relevancy scoring and performance
• Custom field analyzers for tokenization and stemming
• ‘Related to’ or ‘More like this’ querying
• Quick completion suggestions
• Complicated wildcard searching
• Easy Node.js integration
44