• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)
 

Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)

on

  • 8,644 views

Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)

Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.)

Statistics

Views

Total Views
8,644
Views on SlideShare
5,794
Embed Views
2,850

Actions

Likes
9
Downloads
92
Comments
0

7 Embeds 2,850

http://learnmongo.com 2723
http://lanyrd.com 112
http://seattlecoder.com 8
http://webcache.googleusercontent.com 3
http://www.seattlecoder.com 2
http://dashboard.theapp.co 1
http://web.archive.org 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.) Beyond Logging: Using MongoDB to Power a Private Social Network (Oh, and log millions of e-mails too.) Presentation Transcript

    • Building a Private Social Network with MongoDB
      Justin Jenkins@LearnMongo
    • Learn Mongo (.com that is)
      Twitter: @LearnMongo
      MongoDB resources aimed at not scaring off the beginner.
    • Saddleback Church
      21,000+ per weekend @ 10 locations.4,000 online.
      4,300 “small groups”
      20k – 32k people
      ~1 million user records
      3.2 million e-mails a week
      Service trips to ~180 countries
      Food bank serving Orange County
    • Features, easy of use …
    • Project Goals
      Provide a private way for groups to interact and share online.
      Tie into other social networks but keep group data private.
      Provide groups with video & audio resources for growth.
      Create a simple & fast user experience.
    • MongoDB SQL Server
      Solid, durable
      Years of use
      Very easy to query
      Uh Ohs
      Not optimized for the web
      Can be complex to scale
      Stringent schema
      Flexible schema
      Fairly easy to scale
      Optimized for read / inserts
      Uh Ohs
      New, little past use
      Can be harder to query
      Different data format
    • Playing nice with SQL Server
    • Playing nice with SQL Server
      Keep users accounts in SQL
      Use the same user id’s in both SQL and MongoDB.
      Store app specific user data in MongoDB.
      Keep group data and hierarchy in SQL
      Extract group data from SQL when any user of the group logs in, compare.
      Push any changes back to SQL in the background.
    • ProblemSolved?
    • The Problems MongoDB Solved
      Flexible schema design
      Each feed item can have totally different attributes.
      New features can be added quickly.
      Reduces impact on production SQL Server
      Focus server resources on internal staff.
      BSON to JSON
      Ideal for a highly AJAX / JavaScript site.
    • The Feed
    • The Feed
      The main feature of the site is “the feed” where the group can interact and share discussions, prayers, videos, etc.
      We need something very flexible …
      Discussions
      Videos
      Events
      Meetings
      Prayers
      Web Links
      Pictures
    • Schema Design
      Flexible Schema
      Less clutter.
      Cleaner, leaner storage.
      Simpler queries.
      JOINs largely unneeded.
      BSON Format
      Converts into JSON with very little effort to make JavaScript / AJAX centric apps happy.
      Move from the client to the database, back to the client easily.
    • {
      "_id" : "4d19fb939ac0900274000005",
      "_t" : "Discussion",
      "created" : "Tue, 28 Dec 2010 07:00:35 GMT",
      "GroupID" : 129242,
      "eIndividualID" : "22872F9",
      "firstName" : "Bill",
      "lastName" : "Finch",
      "discussionBody" : "I've been not feeling well lately, I hope it's nothing major.",
      "comments": [
      {
      "_id" : "4d19fe329ac090027400000a",
      "eIndividualID" : "FFE4251",
      "created" : "2010-12-28T15:11:46.6760000Z",
      "firstName" : "Robin",
      "lastName" : "Tally",
      "commentText" : "I hope that everything is OK, whatever it is!"
      },
      {
      "_id" : "4d1df3b29ac0900d64000003",
      ... } ]
      }
    • Models (the pretty kind)
    • using System;
      using System.Collections.Generic;
      using System.Linq;
      using System.Web;
      using Newtonsoft.Json;
      using MySmallGroup.Helpers.JsonConverters;
      namespace MySmallGroup.Models
      {
      public abstract class Feed : IFeed, IUserCreatable
      {
      [JsonConverter(typeof(MongoOidConverter))]
      public MongoDB.Oid Id { get; set; }
      public DateTime created { get; set; }
      public intGroupID { get; set; }
      public string eIndividualID { get; set; }
      public string firstName { get; set; }
      public string lastName { get; set; }
      public abstract string shortDescription { get; }
      public intcommentCount { get; set; }
      public IList<comment> comments { get; set; }
      public abstract void ReceivedJsonEncode();
      public Feed Get()
      {
      return this;
      }
      }
      public interface IFeed : IUserCreatable
      {
      MongoDB.Oid Id { get; set; }
      intGroupID { get; set; }
      string eIndividualID { get; set; }
      string firstName { get; set; }
      string lastName { get; set; }
      string shortDescription { get; }
      intcommentCount { get; set; }
      IList<comment> comments { get; set; }
      void ReceivedJsonEncode();
      Feed Get();
      }
      }
    • using System;
      using System.Collections.Generic;
      using System.Linq;
      using System.Web;
      using MySmallGroup.Helpers.ExtensionMethods;
      namespace MySmallGroup.Models
      {
      public class Discussion : Feed
      {
      public string discussionBody { get; set;}
      public Discussion()
      {
      this.commentCount = new int();
      this.comments = new List<comment>();
      this.created = DateTime.Now;
      }
      public override void ReceivedJsonEncode()
      {
      discussionBody = discussionBody.ReceivedJsonEncode();
      }
      public override string shortDescription
      {
      get { return discussionBody; }
      }
      }
      }
    • Live Chat & Video
    • Live Chat & Live Video
      {
      "_id": "4cd261549ac0900c24000006",
      "senderEIndividualID": "1B0D6E8",
      "senderFirstName": "Justin",
      "senderLastName": "Jenkins",
      "sendDate": "Thu, 04 Nov 2010 00:31:32 GMT",
      "groupID": 119420,
      "message": "way faster then devtho"
      }
      {
      "_id": "4cd261a69ac0900c2400000a",
      "senderEIndividualID": "B656641",
      "senderFirstName": "Steve",
      "senderLastName": "Chen",
      "sendDate": "Thu, 04 Nov 2010 00:32:54 GMT",
      "groupID": 119420,
      "message": "oh"
      }
      {
      "_id": "4cd2621b9ac0900c2400001f",
      "senderEIndividualID": "1B0D6E8",
      "senderFirstName": "Justin",
      "senderLastName": "Jenkins",
      "sendDate": "Thu, 04 Nov 2010 00:34:51 GMT",
      "groupID": 119420,
      "message": "I think it&#39;s the facebook pic reload voodoo Jeff made"
      }
    • User Settings
    • User Settings
      {
      "eIndividualID": "A0BE87A",
      "_id": "4cd332389ac0900c2400001c", "ChatTimestamps": false,
      "GroupSettings": [
      {
      "GroupId": 119420,
      "FeedNotificationEmail": false,
      "CommentNotificationEmail": false
      },
      {
      "GroupId": 128834,
      "FeedNotificationEmail": true,
      "CommentNotificationEmail": false
      },
      {
      "GroupId": 125680,
      "FeedNotificationEmail": true,
      "CommentNotificationEmail": true
      }
      ]
      }
    • Email Tracking …
    • E-mail Tracking (the boring stuff.)
      Emails Sent
      350,000 normal e-mails sent a week.
      460,000 daily subscription e-mails sent a day.
      Tracking Needs
      Track each e-mail open, per individual.
      Date
      User Agent
      IP
      If a user opens an e-mail more than once, don’t double count but still track.
      Deduce how quickly users open an e-mail and how many don’t open the e-mail at all.
    • Email Tracking
      {
      "EmailLogID": 14,
      "EmailLogDocumentID": "4d12a55e00ae2611f8000002",
      "EIndividualID": “ZZZZZZZ",
      "Email": "jxxx@zzzzzzzzz.com",
      "SendDate": "Wed, 22 Dec 2010 17:26:54 GMT",
      "_id": "4d12a56a18bee80a7c00243d",
      "Opens": [
      {
      "OpenDate": "2010-12-23T01:59:45",
      "UserAgent": "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1like Mac OS X; en-us) ...",
      "IP": "72.111.250.111"
      },
      {
      "OpenDate": "2010-12-23T16:22:19",
      "UserAgent": "Mozilla/5.0 (Macintosh; U; Intel Mac OS X10_6_5; en-us) ...",
      "IP": "70.111.66.11"
      }
      ]
      }
    • > db.Recipient.find({ "EmailLogID" : 32, "Opens" : { "$exists" : true } }).count();
      141762
    • LearnMongo.com
      Twitter
      @LearnMongo @MongoQuestion @JustinJenkins