Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)

1. Building a Private Social Network with MongoDB Justin Jenkins@LearnMongo

2. Learn Mongo (.com that is) Twitter: @LearnMongo MongoDB resources aimed at not scaring off the beginner.

3. Saddleback Church 21,000+ per weekend @ 10 locations.4,000 online. 4,300 “small groups” 20k – 32k people ~1 million user records 3.2 million e-mails a week Service trips to ~180 countries Food bank serving Orange County

4. Features, easy of use …

5. Project Goals Provide a private way for groups to interact and share online. Tie into other social networks but keep group data private. Provide groups with video & audio resources for growth. Create a simple & fast user experience.

6. MongoDB SQL Server Solid, durable Years of use Very easy to query Uh Ohs Not optimized for the web Can be complex to scale Stringent schema Flexible schema Fairly easy to scale Optimized for read / inserts Uh Ohs New, little past use Can be harder to query Different data format

7. Playing nice with SQL Server

8. Playing nice with SQL Server Keep users accounts in SQL Use the same user id’s in both SQL and MongoDB. Store app specific user data in MongoDB. Keep group data and hierarchy in SQL Extract group data from SQL when any user of the group logs in, compare. Push any changes back to SQL in the background.

9. ProblemSolved?

10. The Problems MongoDB Solved Flexible schema design Each feed item can have totally different attributes. New features can be added quickly. Reduces impact on production SQL Server Focus server resources on internal staff. BSON to JSON Ideal for a highly AJAX / JavaScript site.

11. The Feed

13. The Feed The main feature of the site is “the feed” where the group can interact and share discussions, prayers, videos, etc. We need something very flexible … Discussions Videos Events Meetings Prayers Web Links Pictures

14. Schema Design Flexible Schema Less clutter. Cleaner, leaner storage. Simpler queries. JOINs largely unneeded. BSON Format Converts into JSON with very little effort to make JavaScript / AJAX centric apps happy. Move from the client to the database, back to the client easily.

15. { "_id" : "4d19fb939ac0900274000005", "_t" : "Discussion", "created" : "Tue, 28 Dec 2010 07:00:35 GMT", "GroupID" : 129242, "eIndividualID" : "22872F9", "firstName" : "Bill", "lastName" : "Finch", "discussionBody" : "I've been not feeling well lately, I hope it's nothing major.", "comments": [ { "_id" : "4d19fe329ac090027400000a", "eIndividualID" : "FFE4251", "created" : "2010-12-28T15:11:46.6760000Z", "firstName" : "Robin", "lastName" : "Tally", "commentText" : "I hope that everything is OK, whatever it is!" }, { "_id" : "4d1df3b29ac0900d64000003", ... } ] }

16. Models (the pretty kind)

17. using System; using System.Collections.Generic; using System.Linq; using System.Web; using Newtonsoft.Json; using MySmallGroup.Helpers.JsonConverters; namespace MySmallGroup.Models { public abstract class Feed : IFeed, IUserCreatable { [JsonConverter(typeof(MongoOidConverter))] public MongoDB.Oid Id { get; set; } public DateTime created { get; set; } public intGroupID { get; set; } public string eIndividualID { get; set; } public string firstName { get; set; } public string lastName { get; set; } public abstract string shortDescription { get; } public intcommentCount { get; set; } public IList<comment> comments { get; set; } public abstract void ReceivedJsonEncode(); public Feed Get() { return this; } } public interface IFeed : IUserCreatable { MongoDB.Oid Id { get; set; } intGroupID { get; set; } string eIndividualID { get; set; } string firstName { get; set; } string lastName { get; set; } string shortDescription { get; } intcommentCount { get; set; } IList<comment> comments { get; set; } void ReceivedJsonEncode(); Feed Get(); } }

18. using System; using System.Collections.Generic; using System.Linq; using System.Web; using MySmallGroup.Helpers.ExtensionMethods; namespace MySmallGroup.Models { public class Discussion : Feed { public string discussionBody { get; set;} public Discussion() { this.commentCount = new int(); this.comments = new List<comment>(); this.created = DateTime.Now; } public override void ReceivedJsonEncode() { discussionBody = discussionBody.ReceivedJsonEncode(); } public override string shortDescription { get { return discussionBody; } } } }

19. Live Chat & Video

21. Live Chat & Live Video { "_id": "4cd261549ac0900c24000006", "senderEIndividualID": "1B0D6E8", "senderFirstName": "Justin", "senderLastName": "Jenkins", "sendDate": "Thu, 04 Nov 2010 00:31:32 GMT", "groupID": 119420, "message": "way faster then devtho" } { "_id": "4cd261a69ac0900c2400000a", "senderEIndividualID": "B656641", "senderFirstName": "Steve", "senderLastName": "Chen", "sendDate": "Thu, 04 Nov 2010 00:32:54 GMT", "groupID": 119420, "message": "oh" } { "_id": "4cd2621b9ac0900c2400001f", "senderEIndividualID": "1B0D6E8", "senderFirstName": "Justin", "senderLastName": "Jenkins", "sendDate": "Thu, 04 Nov 2010 00:34:51 GMT", "groupID": 119420, "message": "I think it&#39;s the facebook pic reload voodoo Jeff made" }

22. User Settings

23. User Settings { "eIndividualID": "A0BE87A", "_id": "4cd332389ac0900c2400001c", "ChatTimestamps": false, "GroupSettings": [ { "GroupId": 119420, "FeedNotificationEmail": false, "CommentNotificationEmail": false }, { "GroupId": 128834, "FeedNotificationEmail": true, "CommentNotificationEmail": false }, { "GroupId": 125680, "FeedNotificationEmail": true, "CommentNotificationEmail": true } ] }

24. Email Tracking …

25. E-mail Tracking (the boring stuff.) Emails Sent 350,000 normal e-mails sent a week. 460,000 daily subscription e-mails sent a day. Tracking Needs Track each e-mail open, per individual. Date User Agent IP If a user opens an e-mail more than once, don’t double count but still track. Deduce how quickly users open an e-mail and how many don’t open the e-mail at all.

26. Email Tracking { "EmailLogID": 14, "EmailLogDocumentID": "4d12a55e00ae2611f8000002", "EIndividualID": “ZZZZZZZ", "Email": "jxxx@zzzzzzzzz.com", "SendDate": "Wed, 22 Dec 2010 17:26:54 GMT", "_id": "4d12a56a18bee80a7c00243d", "Opens": [ { "OpenDate": "2010-12-23T01:59:45", "UserAgent": "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1like Mac OS X; en-us) ...", "IP": "72.111.250.111" }, { "OpenDate": "2010-12-23T16:22:19", "UserAgent": "Mozilla/5.0 (Macintosh; U; Intel Mac OS X10_6_5; en-us) ...", "IP": "70.111.66.11" } ] }

27. > db.Recipient.find({ "EmailLogID" : 32, "Opens" : { "$exists" : true } }).count(); 141762

28. LearnMongo.com Twitter @LearnMongo @MongoQuestion @JustinJenkins