Building a Private Social Network with MongoDB<br />Justin Jenkins@LearnMongo<br />
Learn Mongo (.com that is)<br />Twitter: @LearnMongo<br />MongoDB resources aimed at not scaring off the beginner.<br />
Saddleback Church<br />21,000+ per weekend @ 10 locations.4,000 online.<br />4,300 “small groups”<br />20k – 32k people<br...
Features, easy of use …<br />
Project Goals<br />Provide  a private way for groups to interact and share online.<br />Tie into other social networks but...
MongoDB                      SQL Server<br />Solid, durable<br />Years of use<br />Very easy to query<br />Uh Ohs<br />Not...
Playing nice with SQL Server<br />
Playing nice with SQL Server<br />Keep users accounts in SQL<br />Use the same user id’s in both SQL and MongoDB.<br />Sto...
ProblemSolved?<br />
The Problems MongoDB Solved<br />Flexible schema design <br />Each feed item can have totally different attributes.<br />N...
The Feed<br />
The Feed<br />The main feature of the site is “the feed” where the group can interact and share discussions, prayers, vide...
Schema Design<br />Flexible Schema<br />Less clutter.<br />Cleaner, leaner storage.<br />Simpler queries.<br />JOINs large...
{<br />  "_id"            : "4d19fb939ac0900274000005",<br />  "_t"             : "Discussion",<br />  "created"        : ...
Models (the pretty kind)<br />
using System;<br />using System.Collections.Generic;<br />using System.Linq;<br />using System.Web;<br />using Newtonsoft....
using System;<br />using System.Collections.Generic;<br />using System.Linq;<br />using System.Web;<br />using MySmallGrou...
Live Chat & Video<br />
Live Chat & Live Video<br />{<br />  "_id": "4cd261549ac0900c24000006",<br />  "senderEIndividualID": "1B0D6E8",<br />  "s...
User Settings<br />
User Settings<br />{<br />  "eIndividualID": "A0BE87A",<br />  "_id": "4cd332389ac0900c2400001c",  "ChatTimestamps": false...
Email Tracking …<br />
E-mail Tracking (the boring stuff.)<br />Emails Sent<br />350,000 normal e-mails sent a week.<br />460,000 daily subscript...
Email Tracking<br />{<br />  "EmailLogID": 14,<br />  "EmailLogDocumentID": "4d12a55e00ae2611f8000002",<br />  "EIndividua...
> db.Recipient.find({ "EmailLogID" : 32, "Opens" : { "$exists" : true } }).count();<br />141762<br />
LearnMongo.com<br />Twitter<br />@LearnMongo   @MongoQuestion       @JustinJenkins<br />
Upcoming SlideShare
Loading in …5
×

Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)

11,076 views

Published on

Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)

Published in: Technology
0 Comments
12 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
11,076
On SlideShare
0
From Embeds
0
Number of Embeds
2,925
Actions
Shares
0
Downloads
155
Comments
0
Likes
12
Embeds 0
No embeds

No notes for slide

Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)

  1. 1. Building a Private Social Network with MongoDB<br />Justin Jenkins@LearnMongo<br />
  2. 2. Learn Mongo (.com that is)<br />Twitter: @LearnMongo<br />MongoDB resources aimed at not scaring off the beginner.<br />
  3. 3. Saddleback Church<br />21,000+ per weekend @ 10 locations.4,000 online.<br />4,300 “small groups”<br />20k – 32k people<br />~1 million user records<br />3.2 million e-mails a week<br />Service trips to ~180 countries<br />Food bank serving Orange County<br />
  4. 4. Features, easy of use …<br />
  5. 5. Project Goals<br />Provide a private way for groups to interact and share online.<br />Tie into other social networks but keep group data private.<br />Provide groups with video & audio resources for growth.<br />Create a simple & fast user experience.<br />
  6. 6. MongoDB SQL Server<br />Solid, durable<br />Years of use<br />Very easy to query<br />Uh Ohs<br />Not optimized for the web<br />Can be complex to scale<br />Stringent schema <br />Flexible schema<br />Fairly easy to scale<br />Optimized for read / inserts<br />Uh Ohs<br />New, little past use<br />Can be harder to query<br />Different data format<br />
  7. 7. Playing nice with SQL Server<br />
  8. 8. Playing nice with SQL Server<br />Keep users accounts in SQL<br />Use the same user id’s in both SQL and MongoDB.<br />Store app specific user data in MongoDB.<br />Keep group data and hierarchy in SQL<br />Extract group data from SQL when any user of the group logs in, compare.<br />Push any changes back to SQL in the background.<br />
  9. 9. ProblemSolved?<br />
  10. 10. The Problems MongoDB Solved<br />Flexible schema design <br />Each feed item can have totally different attributes.<br />New features can be added quickly.<br />Reduces impact on production SQL Server<br />Focus server resources on internal staff.<br />BSON to JSON<br />Ideal for a highly AJAX / JavaScript site.<br />
  11. 11. The Feed<br />
  12. 12.
  13. 13. The Feed<br />The main feature of the site is “the feed” where the group can interact and share discussions, prayers, videos, etc.<br />We need something very flexible …<br />Discussions<br />Videos<br />Events<br />Meetings<br />Prayers<br />Web Links<br />Pictures<br />
  14. 14. Schema Design<br />Flexible Schema<br />Less clutter.<br />Cleaner, leaner storage.<br />Simpler queries.<br />JOINs largely unneeded.<br />BSON Format<br />Converts into JSON with very little effort to make JavaScript / AJAX centric apps happy.<br />Move from the client to the database, back to the client easily.<br />
  15. 15. {<br /> "_id" : "4d19fb939ac0900274000005",<br /> "_t" : "Discussion",<br /> "created" : "Tue, 28 Dec 2010 07:00:35 GMT",<br /> "GroupID" : 129242,<br /> "eIndividualID" : "22872F9",<br /> "firstName" : "Bill",<br /> "lastName" : "Finch",<br /> "discussionBody" : "I've been not feeling well lately, I hope it's nothing major.",<br /> "comments": [<br /> {<br /> "_id" : "4d19fe329ac090027400000a",<br /> "eIndividualID" : "FFE4251",<br /> "created" : "2010-12-28T15:11:46.6760000Z",<br /> "firstName" : "Robin",<br /> "lastName" : "Tally",<br /> "commentText" : "I hope that everything is OK, whatever it is!"<br /> },<br /> {<br /> "_id" : "4d1df3b29ac0900d64000003",<br />... } ]<br />}<br />
  16. 16. Models (the pretty kind)<br />
  17. 17. using System;<br />using System.Collections.Generic;<br />using System.Linq;<br />using System.Web;<br />using Newtonsoft.Json;<br />using MySmallGroup.Helpers.JsonConverters;<br />namespace MySmallGroup.Models<br />{<br /> public abstract class Feed : IFeed, IUserCreatable<br /> {<br /> [JsonConverter(typeof(MongoOidConverter))]<br /> public MongoDB.Oid Id { get; set; }<br />public DateTime created { get; set; }<br /> public intGroupID { get; set; }<br /> public string eIndividualID { get; set; }<br /> public string firstName { get; set; }<br /> public string lastName { get; set; }<br /> public abstract string shortDescription { get; }<br /> public intcommentCount { get; set; }<br /> public IList<comment> comments { get; set; }<br /> public abstract void ReceivedJsonEncode();<br /> public Feed Get()<br /> {<br /> return this;<br /> }<br /> }<br /> public interface IFeed : IUserCreatable<br /> {<br />MongoDB.Oid Id { get; set; }<br />intGroupID { get; set; }<br /> string eIndividualID { get; set; }<br /> string firstName { get; set; }<br /> string lastName { get; set; }<br /> string shortDescription { get; }<br />intcommentCount { get; set; }<br />IList<comment> comments { get; set; }<br /> void ReceivedJsonEncode();<br /> Feed Get();<br /> }<br />}<br />
  18. 18. using System;<br />using System.Collections.Generic;<br />using System.Linq;<br />using System.Web;<br />using MySmallGroup.Helpers.ExtensionMethods;<br />namespace MySmallGroup.Models<br />{<br /> public class Discussion : Feed<br /> {<br /> public string discussionBody { get; set;}<br /> public Discussion()<br /> {<br />this.commentCount = new int();<br />this.comments = new List<comment>();<br />this.created = DateTime.Now;<br /> }<br /> public override void ReceivedJsonEncode()<br /> {<br />discussionBody = discussionBody.ReceivedJsonEncode();<br /> }<br /> public override string shortDescription<br /> {<br /> get { return discussionBody; }<br /> }<br /> }<br />}<br />
  19. 19. Live Chat & Video<br />
  20. 20.
  21. 21. Live Chat & Live Video<br />{<br /> "_id": "4cd261549ac0900c24000006",<br /> "senderEIndividualID": "1B0D6E8",<br /> "senderFirstName": "Justin",<br /> "senderLastName": "Jenkins",<br /> "sendDate": "Thu, 04 Nov 2010 00:31:32 GMT",<br /> "groupID": 119420,<br /> "message": "way faster then devtho"<br />}<br />{<br /> "_id": "4cd261a69ac0900c2400000a",<br /> "senderEIndividualID": "B656641",<br /> "senderFirstName": "Steve",<br /> "senderLastName": "Chen",<br /> "sendDate": "Thu, 04 Nov 2010 00:32:54 GMT",<br /> "groupID": 119420,<br /> "message": "oh"<br />} <br />{<br /> "_id": "4cd2621b9ac0900c2400001f",<br /> "senderEIndividualID": "1B0D6E8",<br /> "senderFirstName": "Justin",<br /> "senderLastName": "Jenkins",<br /> "sendDate": "Thu, 04 Nov 2010 00:34:51 GMT",<br /> "groupID": 119420,<br /> "message": "I think it's the facebook pic reload voodoo Jeff made"<br />}<br />
  22. 22. User Settings<br />
  23. 23. User Settings<br />{<br /> "eIndividualID": "A0BE87A",<br /> "_id": "4cd332389ac0900c2400001c", "ChatTimestamps": false,<br /> "GroupSettings": [<br /> {<br /> "GroupId": 119420,<br /> "FeedNotificationEmail": false,<br /> "CommentNotificationEmail": false<br /> },<br /> {<br /> "GroupId": 128834,<br /> "FeedNotificationEmail": true,<br /> "CommentNotificationEmail": false<br /> },<br /> {<br /> "GroupId": 125680,<br /> "FeedNotificationEmail": true,<br /> "CommentNotificationEmail": true<br /> }<br /> ]<br />}<br />
  24. 24. Email Tracking …<br />
  25. 25. E-mail Tracking (the boring stuff.)<br />Emails Sent<br />350,000 normal e-mails sent a week.<br />460,000 daily subscription e-mails sent a day.<br />Tracking Needs<br />Track each e-mail open, per individual.<br />Date<br />User Agent<br />IP<br />If a user opens an e-mail more than once, don’t double count but still track.<br />Deduce how quickly users open an e-mail and how many don’t open the e-mail at all.<br />
  26. 26. Email Tracking<br />{<br /> "EmailLogID": 14,<br /> "EmailLogDocumentID": "4d12a55e00ae2611f8000002",<br /> "EIndividualID": “ZZZZZZZ",<br /> "Email": "jxxx@zzzzzzzzz.com",<br /> "SendDate": "Wed, 22 Dec 2010 17:26:54 GMT",<br /> "_id": "4d12a56a18bee80a7c00243d",<br />"Opens": [<br />{<br /> "OpenDate": "2010-12-23T01:59:45",<br /> "UserAgent": "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1like Mac OS X; en-us) ...",<br />"IP": "72.111.250.111"<br /> },<br />{<br /> "OpenDate": "2010-12-23T16:22:19",<br /> "UserAgent": "Mozilla/5.0 (Macintosh; U; Intel Mac OS X10_6_5; en-us) ...",<br />"IP": "70.111.66.11"<br /> }<br /> ]<br />}<br />
  27. 27. > db.Recipient.find({ "EmailLogID" : 32, "Opens" : { "$exists" : true } }).count();<br />141762<br />
  28. 28. LearnMongo.com<br />Twitter<br />@LearnMongo @MongoQuestion @JustinJenkins<br />

×