MongoSV Schema Workshop

Schema Design
Workshop
Sridhar Nanjundeswaran

Software Engineer, 10Gen
sridhar@10gen.com
@snanjund

Wednesday, December 5, 12

Agenda

• Part One - Basic Schema & Patterns
• Part Two - Schema Design
• Part Three - Sharding
• Part Four: - Replication


Why is schema design
different?
• RDBMS design you ask "what answers do I have"
• MongoDB you ask "what questions will I have"


Goals

• Learn Data Modeling with MongoDB
• Labs to try to solve problems
• Understand implications of
• Replication
• Sharding

Please, ask many, many questions!


Part One
Basic Schema & Patterns


So why model data?

http://bit.ly/SSs7QB


Normalization
• 1970 E.F.Codd introduces 1st Normal Form (1NF)
• 1971 E.F.Codd introduces 2nd and 3rd Normal Form (2NF, 3NF)
• 1974 Codd & Boyce deﬁne Boyce/Codd Normal Form (BCNF)
• 2002 Date, Darween, Lorentzos deﬁne 6th Normal Form (6NF)

Goals:
• Avoid anomalies when inserting, updating or deleting
• Minimize redesign when extending the schema
• Make the model informative to users
• Avoid bias towards a particular style of query

* source : wikipedia

So today’s example will use...

http://bit.ly/RyIOvO


Terminology
RDBMS MongoDB
Table Collection
Row(s) JSON
Document
Index Index
Join Embedding
&
Linking
Partition Shard
Partition
Key Shard
Key


Schema Design
Relational Database


Schema Design
MongoDB


Schema Design
MongoDB
linking


Schema Design embedding
MongoDB
linking


Basic schema

Design documents that simply map to your application

> post = { author: "Hergé",
date: ISODate("2011-09-18T09:56:06.298Z"),
text: "Destination Moon",
tags: ["comic", "movie"]
}

> db.blogs.save(post)


Find the document
> db.blogs.find()

{ _id: ObjectId("4c4ba5c0672c685e5e8aabf3"),
author: "Hergé",
date: ISODate("2011-09-18T09:56:06.298Z"),
text: "Destination Moon",
tags: [ "comic", "movie" ]
}

Notes:
• ID must be unique, but can be anything you’d like
• MongoDB will generate a default ID if one is not
supplied


Add an index, ﬁnd via Index

Secondary index for “author”

// 1 means ascending, -1 means descending
> db.blogs.ensureIndex( { author: 1 } )

> db.blogs.find( { author: 'Hergé' } )

{ _id: ObjectId("4c4ba5c0672c685e5e8aabf3"),
date: ISODate("2011-09-18T09:56:06.298Z"),
author: "Hergé",
... }


Examine the query plan

> db.blogs.find( { author: "Hergé" } ).explain()
{
! "cursor" : "BtreeCursor author_1",
! "nscanned" : 1,
! "nscannedObjects" : 1,
! "n" : 1,
! "millis" : 5,
! "indexBounds" : {
! ! "author" : [
! ! ! [
! ! ! ! "Hergé",
! ! ! ! "Hergé"
! ! ! ]
! ! ]
! }
}


Examine the query plan

> db.blogs.find( { author: "Hergé" } ).explain()
{
! "cursor" : "BtreeCursor author_1",
! "nscanned" : 1,
! "nscannedObjects" : 1,
Number of objects
! "n" : 1, returned
! "millis" : 5,
! "indexBounds" : { How long it took
! ! "author" : [
! ! ! [
! ! ! ! "Hergé",
! ! ! ! "Hergé"
! ! ! ]
! ! ]
! }
}


Query operators
Conditional operators:
$ne, $in, $nin, $mod, $all, $size, $exists, $type, ..
$lt, $lte, $gt, $gte, $ne...

// find posts with any tags
> db.blogs.find( { tags: { $exists: true } } )

Regular expressions:
// posts where author starts with h
> db.blogs.find( { author: /^h/i } )

Counting:
// number of posts written by Hergé
> db.blogs.find( { author: "Hergé" } ).count()


Extending the Schema

http://bit.ly/PpjT1l


> new_comment =
{ author: "Kyle",
date: new Date(),
text: "great book" }

> db.blogs.update(
{ text: "Destination Moon" },
{ "$push": { comments: new_comment },
"$inc": { comments_count: 1 }
} )


> new_comment =
{ author: "Kyle",
date: new Date(),
text: "great book" }

> db.blogs.update(
{ text: "Destination Moon" },
{ "$push": { comments: new_comment },
"$inc": { comments_count: 1 }
} )

Add element to
Increment counter array


> db.blogs.find( { author: "Hergé"} )

{ _id : ObjectId("4c4ba5c0672c685e5e8aabf3"),
author : "Hergé",
date : ISODate("2011-09-18T09:56:06.298Z"),
text : "Destination Moon",
tags : [ "comic", "movie" ],
comments : [
! {
! ! author : "Kyle",
! ! date : ISODate("2011-09-19T09:56:06.298Z"),
! ! text : "great book"
! }
],
comments_count: 1
}


// create index on nested documents:
> db.blogs.ensureIndex( { "comments.author": 1 } )

> db.blogs.find( { "comments.author": "Kyle" } )

// find last 5 posts:
> db.blogs.find().sort( { date: -1 } ).limit(5)

// most commented post:
> db.blogs.find().sort( { comments_count: -1 } ).limit(1)

When sorting, check if you need an index


Common Patterns

http://bit.ly/SNnt4z


Inheritance

http://bit.ly/T7MqUz


Inheritance


Single Table Inheritance -
RDBMS
select * from shapes;

id type area radius length width

1 circle 3.14 1

2 square 4 2

3 rect 10 5 2


MongoDB
> db.shapes.find()
{ _id: "1", type: "c", area: 3.14, radius: 1}
{ _id: "2", type: "s", area: 4, length: 2}
{ _id: "3", type: "r", area: 10, length: 5, width: 2}

missing values not
stored!


MongoDB
> db.shapes.find()

// find shapes where radius > 0
> db.shapes.find( { radius: { $gt: 0 } } )


MongoDB
> db.shapes.find()

// find shapes where radius > 0
> db.shapes.find( { radius: { $gt: 0 } } )

// create index
> db.shapes.ensureIndex( { radius: 1 }, { sparse:true } )

index only values
present!


One to Many

http://bit.ly/Oqbt8z


One to Many

One to Many relationships can specify
• degree of association between objects
• containment
• life-cycle


One to Many
Embedded Array
•$slice operator to return subset of comments
•some queries harder
•e.g ﬁnd latest comments across all blogs
blogs: {
author : "Hergé",
date : ISODate("2011-09-18T09:56:06.298Z"),
comments : [
! { author : "Kyle",
! ! date : ISODate("2011-09-19T09:56:06.298Z"),
! ! text : "great book" }
] }

> db.blogs.find( { author: "Hergé" },
{ comment: { $slice : 10 } } )


One to Many
Normalized (2 collections)
• most ﬂexible
• more queries

blogs: { _id: 1000,
author: "Hergé",
date: ISODate("2011-09-18T09:56:06.298Z"),
comments: [
! {comment : 1)}
]}

comments : { _id : 1,
blog: 1000,
author : "Kyle",
! ! date : ISODate("2011-09-19T09:56:06.298Z")}

> blog = db.blogs.find( { text: "Destination Moon" } );
> db.comments.find( { blog: blog._id } ).limit(5);


Many to Many

http://bit.ly/QTzhBF


Many - Many

Example:

• Blog can have many Tags
• Tag can be used by many Blogs


Many - Many
// Each Tag lists the "_id" of the Blog
tags:
{ _id: 20,
name: "comic", // Unique
blog_ids: [ 10, 11, 12 ] }

{ _id: 30,
name: "movie", // Unique
blog_ids: [ 10 ] }


Many - Many
tags:
{ _id: 20,
blog_ids: [ 10, 11, 12 ] }

{ _id: 30,
blog_ids: [ 10 ] }

// Each Blog lists the "tag" of the Tags
blogs:
{ _id: 10, name: "Destination Moon",
tags: [ "comic", "movie" ] }


Many - Many
tags:
{ _id: 20,
blog_ids: [ 10, 11, 12 ] }
links via unique key, in this
{ _id: 30, case "tags", could be "_id"
blog_ids: [ 10 ] }

blogs:


Many - Many
tags:
{ _id: 20,
blog_ids: [ 10, 11, 12 ] }

{ _id: 30,
blog_ids: [ 10 ] }

blogs:

// All Tags for a given Blog
> db.tags.find( { blog_ids: 10 } )


Use _id or not?

blogs: blogs:
{ _id: 10, name: "..." { _id: 10, name: "..."
tags: [ "comic", "movie" ] tags: [ 10, 20 ]
} }

Pros: Pros:
• Single query • Single update
Cons: Cons:
• Cascade any changes • Second query required


Alternative
// Each Blog lists the _id of the Tag
blogs:
tag_ids: [ 20, 30 ] }

// Association not stored on the Tag
tags:
{ _id: 20,
name: "comic" }


Alternative
blogs:
tag_ids: [ 20, 30 ] }

tags:
{ _id: 20,
name: "comic" }

// All Blogs for a given Tag
> db.blogs.find( { tag_ids: 20 } )


Alternative
blogs:
tag_ids: [ 20, 30 ] }

tags:
{ _id: 20,
name: "comic" }

// All Blogs for a given Tag
> db.blogs.find( { tag_ids: 20 } )

// All Tags for a given Blog
> blog = db.blogs.findOne( { _id: 10 } )
> db.tags.find({_id: {$in : blog.tag_ids}})


Many - Many
Intersection Attributes
Example:

• Blog can have many Tags
• Tag can be used my many Blogs
• When a Tag is used, record the usage date


Many - Many
Normalized
blogs: { _id: 10, name: "...", tag_ids: [ 20, 30 ] }

tags: { _id: 20, name: "comic" }

// Store the interaction and usage date
usages: { blog_id: 10, // Blog _id
tag_id : 20, // Tag _id
usage: ISODate("2012-10-12...") }

// Find the Tags for a Blog
for(var c = db.usages.find({ blog_id: 10 });
c.hasNext(); )
{ u = c.next();
t = db.tags.findOne( { _id: c.tag_id } )
printjson( u.usage );


Many - Many
Intersection Attributes
// Each Blog lists the Blog Usage Object
blogs:
tags: [
{ tag: "comic", usage: ISODate("2012-10-12...") }
{ tag: "movie", usage: ISODate("2012-09-11...") }
] }

// Find the Tags for a Blog
> db.blogs.find( { _id: 10 }, { tags: 1} )

Pros:
• Usage object encapsulated where used
Cons:
• If updates allowed, changes will have to be cascaded


Summary

• Single biggest performance factor
• More choices than in an RDBMS
• Embedding, index design, shard keys


Part Two
Schema Design


Lab #1
Design Schema for Twitter

• Model each users activity stream
• Users
• Name, email address, display name
• Tweets
• Text
• Who
• Timestamp


Lab #1 - Solution A
Two Collections
// users - one doc per user
{ _id: "alvin",
email: "alvin@10gen.com",
display: "jonnyeight"
}

// tweets - one doc per user per tweet
{
user: "bob",
for: "alvin",
tweet: "20111209-1231",
text: "Best Tweet Ever!",
ts: ISODate("2011-09-18T09:56:06.298Z")
}


Lab #1 - Solution B
Embedded Tweets
// users - one doc per user with all tweets
{ _id: "alvin",
display; "jonnyeight",
tweets: [
! {
! ! user: "bob",
! ! tweet: "20111209-1231",
! ! text: "Best Tweet Ever!",
ts: ISODate("2011-09-18T09:56:06.298Z")
! }
]
}


Embedding
• Great for read performance
• One seek to load entire object
• One roundtrip to database
• Writes can be slow if adding to objects all the time


Linking or Embedding?

Linking can make some queries easy

// Find latest 50 tweets for "alvin"
> db.tweets.find( { _id:"alvin"}
)
.sort( {ts:-1} )
.limit(50)

But what effect does this have on the systems?


Collection 1

Index 1


Collection 1 Virtual
Address
Space 1

Index 1 This is your virtual
memory size
(mapped)


Collection 1 Virtual
Address
Space 1

Physical
RAM

Index 1

This is your
resident
memory size


Collection 1 Virtual Disk
Address
Space 1

Physical
RAM

Index 1


Address
Space 1

Physical
RAM

Index 1

100 ns
=
10,000,000 ns
=


Address
Space 1

Physical
RAM

Index 1

1

2
> db.tweets.find( { _id: "alvin" } )
.sort( { ts: -1 } )
.limit(10) 3

Linking = Many seeks + random reads


Address
Space 1

Physical
RAM

Index 1

> db.tweets.find( { _id: "alvin" } )

1

Embedding = Large Sequential Read


Lab #2
Alternative Schema

• Display last 10 tweets from today
• Efficiently use memory and Disk seeks / IOPs


Lab #2 - Solution
Buckets
// tweets : one doc per user per day
> db.tweets.findOne()

{
_id: "alvin-2011/12/09",
tweets: [
{ user: "Bob",
! tweet: "20111209-1231",
! text: "Best Tweet Ever!" } ,
! { author: "Joe",
! tweet: "20111210-9025",
! date: "May 27 2011",
! text: "Stuck in traffic (again)" }
]
}


Lab #2 - Solution
Last 10 Tweets
> db.tweets.find( { _id: "alvin-2011/12/09" },
{ tweets: { $slice : 10 } }
)
.sort( { _id: -1 } )
.limit(1)


Lab #2 - Solution
Adding a Tweet
> tweet = { user: "Bob",
! tweet: "20111209-1231",
! text: "Best Tweet Ever!" }

> db.tweets.update( { _id : "alvin-2011/12/09" },
{ $push : { tweets : tweet } );


Lab #2 - Solution
Getting All Tweets
> cursor = db.tweets.find
( { _id : /^alvin/ } ).sort( { _id : -1 } )

> while ( cursor.hasNext() ) {
doc = cursor.next();
for ( var i=0; i<doc.tweets.length; i++ )
printjson( doc.tweets[i] )
}


Lab #2 - Solution
Deleting a Tweet
> db.tweets.update(
{ _id: "alvin-20111209" },
{ $pull: { tweets: { tweet: "20111209-1231" } }
)


Address
Space 1

Physical
RAM

Index 1

> db.tweets.find( { _id: "alvin-2011/12/09" },
{ tweets: { $slice : 10 } } ) 1
.sort( { _id: -1 } )
.limit(1)

Bucket = 1 seek + 1 sequential read


Trees

http://bit.ly/Oqc8Xs


Trees

Hierarchical information


Trees

Full Tree in Document

{ retweet: [
{ who: “Kyle”, text: “...”,
retweet: [
{who: “James”, text: “...”,
retweet: []}
]}
]
}

Pros: Single Document, Performance, Intuitive

Cons: Hard to search, Partial Results, 16MB limit


Array of Ancestors A B C

// Store all Ancestors of a node E D
{ _id: "a" }
{ _id: "b", tree: [ "a" ], retweet: "a" } F
{ _id: "c", tree: [ "a", "b" ], retweet: "b" }
{ _id: "d", tree: [ "a", "b" ], retweet: "b" }
{ _id: "e", tree: [ "a" ], retweet: "a" }
{ _id: "f", tree: [ "a", "e" ], retweet: "e" }


Array of Ancestors A B C

// Store all Ancestors of a node E D
{ _id: "a" }
{ _id: "b", tree: [ "a" ], retweet: "a" } F
{ _id: "c", tree: [ "a", "b" ], retweet: "b" }
{ _id: "d", tree: [ "a", "b" ], retweet: "b" }
{ _id: "e", tree: [ "a" ], retweet: "a" }
{ _id: "f", tree: [ "a", "e" ], retweet: "e" }

// find all direct retweets of "b"
> db.tweets.find( { retweet: "b" } )

// find all retweets of "e" anywhere in tree
> db.tweets.find( { tree: "e" } )

// find tweet history of f:
> tweets = db.tweets.findOne( { _id: "f" } ).tree
> db.tweets.find( { _id: { $in : tweets } } )


Trees as Paths A B C

E D
Store hierarchy as a path expression
• Separate each node by a delimiter, e.g. “/” F
• Use text search for ﬁnd parts of a tree
{ retweets: [
{ _id: "a", text: "initial tweet",
path: "a" },
{ _id: "b", text: "reweet with comment",
path: "a/b" },
{ _id: "c", text: "reply to retweet",
path : "a/b/c"} ] }

// Find the conversations "a" started
> db.tweets.find( { path: /^a/i } )


Queues & Workﬂows

http://bit.ly/QeNsPX


Lab #3
Following Requests
• Users are allowed to "follow" another user
• User send a "follow" request
• Follower approves or not
• Requests are timed out after 7 days
• The approval is an async process


Lab #3 - Solution
Queues & Workﬂows
• Need to maintain order and state
• Ensure that updates are atomic
> db.approvals.insert(
{ inprogress: false,
approved: false,
priority: 1,
text: "Hey Jim, want to follow you!"
} );
// find highest priority approval and mark as in-progress
job = db.approvals.findAndModify({
query: { inprogress: false },
sort: { priority: -1 },
update: { $set: { inprogress: true,
started: new Date() } },
new: true})


Lab #3 - Solution
Queues & Workﬂows
updated

{ inprogress: true,
priority: 1,
approved: False,
started: ISODate("2011-09-18T09:56:06.298Z")
...
}
added


Lab #3 - Solution
Queues & Workﬂows
• Follower approves request
// update approval after receiving approval
> job = db.approvals.update(
{ _id: "1234" },
{ $set: { approved: true } } )

• System times out request after 7 days
var limit=new Date();
limit.setDate(limit.getDate()-7);

> job = db.approvals.update(
{ inprogress: true,
started: { $gt: limit} },
{ $set: { approved: false } } )


Lab #4
Voting

Twitter meets Stack Overﬂow

• Users can "vote" for a tweet
• A user can "vote" once and only once
• Need to display current votes


Lab #4 - Solution
Votes
// One document per voter per tweet
> db.votes.insert(
{ tweet: "20111209-1231",
voter: "alvin"
} );

// Unique index guarantees the user can't vote twice
> db.votes.ensureIndex( { tweet: 1, voter: 1 },
{ unique: true } );

// Count will return the number of votes cast
> db.votes.find({ tweet: "20111209-1231" }).count()


Count or Not?

• Indexes in MongoDB are not counting
• The count has to be computed via a index scan
// One summary document per tweet, no "voter" key
> db.votes.update(
{ tweet: "20111209-1231",
voter: { $exists: false } },
{ "$inc": { count: 1 } },
true, false );

// Return the count for the no "voter" document
> db.votes.find( { tweet: "20111209-1231",
voter: { $exists: false } },
{ count: 1, _id: 0} )


Lab #5
Time Series
• Records votes by
• Day, Hour, Minute
• Show time series of votes cast


Lab #5 - Solution A
Time Series
// Time series buckets, hour and minute sub-docs
{ _id: "20111209-1231",
ts: ISODate("2011-12-09T00:00:00.000Z")
daily: 67,
hourly: { 0: 23, 1: 14, 2: 19 ... 23: 72 },
minute: { 0: 0, 1: 4, 2: 6 ... 1439: 0 }
}


Lab #5 - Solution A
Time Series
// Add one to the last minute before midnight
> db.votes.update(
{ _id: "20111209-1231",
ts: ISODate("2011-12-09T00:00:00.037Z") },
{ $inc: { daily: 1 },
$inc: { "hourly.23": 1 },
$inc: { "minute.1439": 1 } )

What is the cost of updating the minute before
midnight?


BSON Storage

• Sequence of key/value pairs
• NOT a hash map
• Optimized to scan quickly

0 1 2 3 ... 1439

• 1439 skips


BSON Storage
• Can skip sub-documents

0 1 ... 23
1 ... 59 60 ... 119 1380 ... 1439

• 23 skips (hours) + 59 skips (minutes) = 82 skips


Lab #5 - Solution B
Time Series
// Time series buckets, each hour a sub-document
{ _id: "20111209-1231",
ts: ISODate("2011-12-09T00:00:00.000Z")
daily: 67,
minute: { 0: { 0: 0, 1: 7, ... 59: 2 },
...
23: { 0: 15, ... 59: 6 } }
}

// Add one to the last second before midnight
> db.votes.update(
{ _id: "20111209-1231" },
ts: ISODate("2011-12-09T00:00:00.000Z") },
{ $inc: { daily: 1 },
$inc: { "minute.23.59": 1 } })


Lab #6
Inventory

• User has a number of "votes" they can use


Lab #6 - Solution
Inventory
// Number of votes and who voted for
{ _id: "alvin",
votes: 42,
voted_for: []
}

// Subtract a vote and add the voted for tweet
// "20111209-1231"
> db.user.update(
{ _id: "alvin",
votes : { $gt : 0},
voted_for: { $ne: "20111209-1231" }},

{ "$push": { voted_for: "20111209-1231"},
"$inc": { votes: -1}
} )


Lab #6 - Solution
Inventory
// After vote
decremented
> db.votes.findOne()
{ _id: "alvin",
votes: 41,
voted_for: ["20111209-1231"]
}

added


Lab #7
Statistic Buckets
• Record referring web sites on customer sign up
• Independent counter for each web site


Lab #7 - Solution A
Statistic Buckets
{ _id: "alvin",
referrers: [
{ domain: "www.google.co.uk", count: 4 },
{ domain: "www.yahoo.com", count: 1 },
] }


Lab #7 - Solution A
Statistic Buckets
{ _id: "alvin",
referrers: [
] }

> db.referers.update(
{ "referrers.domain": "www.google.co.uk" },
{ $inc: { "referrers.$.count": 1 } } )


Lab #7 - Solution A
Statistic Buckets
{ _id: "alvin",
referrers: [
] }

{ "referrers.domain": "www.google.co.uk" },
{ $inc: { "referrers.$.count": 1 } } )

{ _id: "alvin",
referrers: [
] }


Lab #7 - Solution A
Statistic Buckets

{ "referrers.domain": "www.bing.com" },
{ $inc: {"referrers.$.count": 1 } }, false, true )

What happens if a new referring site is used?


Lab #7 - Solution B
Statistic Buckets
// Need to replace dots with underscores
{ _id: "alvin",
referrers:
{ "www_google_co_uk": 4,
"www_yahoo_com": 1 },
}

// simple $inc will add www_bing_com if not present
{ _id: "alvin" },
{ $inc: { "referrers.www_bing_com": 1 } },
true, false);


Part Three
Sharding


What is Sharding

• Ad-hoc partitioning
• Consistent hashing
• Amazon Dynamo
• Range based partitioning
• Google BigTable
• Yahoo! PNUTS
• MongoDB


MongoDB Sharding

• Automatic partitioning and management
• Range based
• Convert to sharded system with no downtime
• Fully consistent
• No code changes required


Sharding - Range distribution
sh.shardCollection("mydb.tweets",
{_id:
1}
,
false)

shard01 shard02 shard03


Sharding - Range distribution


a-i j-r s-z


Sharding - Splits


a-i ja-jz s-z
k-r


Sharding - Splits


a-i ja-ji s-z
ji-js
js-jw
jz-r


Sharding - Auto Balancing


a-i ja-ji s-z
ji-js
js-jw js-jw
jz-r jz-r


Sharding - Auto Balancing


a-i ja-ji s-z
ji-js
js-jw
jz-r


Sharding for caching


Sharding for caching
96 GB Mem
3:1 Data/Mem

shard01

a-i
300 GB Data

j-r
s-z

300 GB


Aggregate Horizontal Resources
96 GB Mem 96 GB Mem 96 GB Mem
1:1 Data/Mem 1:1 Data/Mem 1:1 Data/Mem


a-i j-r s-z
300 GB Data

j-r
s-z

100 GB 100 GB 100 GB


Sharding Features
• Shard data without no downtime
• Automatic balancing as data is written
• Commands routed (switched) to correct node
• Inserts - must have the Shard Key
• Updates - can have the Shard Key
• Queries
• With Shard Key - routed to nodes
• Without Shard Key - scatter gather
• Indexed / Sorted Queries
• With Shard Key - routed in order
• Without Shard Key - distributed sort merge


Lab #8
Sharding Twitter Pictures

User can upload pictures to Twitter feed

{ photo_id : ???? , data : <binary> }

What should photo_id be?
How will photo_id be sharded?


Lab #8
Sharding Key
{ photo_id : ???? , data : <binary> }

What’s the right key?
• auto increment
• MD5( data )
• month() + MD5( data )


Right balanced access
• Only have to keep small
portion in ram
• Time Based
• Right shard "hot" • ObjectId
• Auto Increment


Random access

• Have to keep entire
index in ram
• All shards "warm"
• Hash


Segmented access

• Have to keep some
index in ram
• Some shards "warm"
•Month + Hash


Lab #9
Single Identities
// Shard by _id
ids:
{ _id : "alvin",
addresses: [ { state : "CA", country: "USA" },
{ country: "UK" } ]
}

How would the following queries be executed?

> db.ids.find( { _id: "alvin"} )
> db.ids.find( { email: "alvin@10gen.com" } )


Sharding - Routed Query
find(
{
_id:
"alvin"}
)


a-i ja-ji s-z
ji-js
js-jw
jz-r


Sharding - Scatter Gather
find(
{
email:
"alvin@10gen.com"
}
)


a-i ja-ji s-z
ji-js
js-jw
jz-r


Lab #9
Multiple Identities

User can have multiple identities
• twitter name
• email address
• facebook name
• etc.
What is the best sharding key & schema design?


Lab #9 - Solution A
Multiple Identities
// Shard by _id
{ _id: "alvin",
fb: "alvin.richards", // facebook
li: "alvin.j.richards", // linkedin
tweets: [ ... ]
}

Lookup by _id hits 1 node
Lookup by email, li or fb is scatter gather
Cannot create a unique index on email, li or fb


Lab #9 - Solution B
Multiple Identities
identities
{ _id: { _id: "alvin"}, info: "1200-42"}
{ _id: { em: "alvin@10gen.com"}, info: "1200-42"}
{ _id: { li: "alvin.j.richards"}, info: "1200-42"}

tweets
{ _id: "1200-42",
tweets: [ ... ]
}

• Shard identities on { _id: 1}
• Can create unique index on _id
• Shard info on { _id: 1 }


Sharding - Multiple Identities

em: a-q em: r-z _id: a-z

_id: "Min"- li: d-r
"1100"
li: s-z _id: "1100"- _id: "1200"-
"1200" "Max"
li: a-c

ids tweets
collection collection

ids.find({
_id:

{"em","alvin@10gen.com
})


_id: "Min"- li: d-r
"1100"
li: s-z _id: "1100"- _id: "1200"-
"1200" "Max"
li: a-c

ids tweets

ids.find({
_id:

{"em","alvin@10gen.com
})

tweets.find({
_id:
"1200-‐42"
})


_id: "Min"- li: d-r
"1100"
li: s-z _id: "1100"- _id: "1200"-
"1200" "Max"
li: a-c

ids tweets

Part Four
Replication


Types of outage
• Planned
• Hardware upgrade
• O/S or ﬁle-system tuning
• Relocation of data to new ﬁle-system / storage
• Software upgrade

• Unplanned
• Hardware failure
• Data center failure
• Region outage
• Human error
• Application corruption


Replica Sets

• Data Protection
• Multiple copies of the data
• Spread across Data Centers, AZs
• High Availability
• Automated Failover
• Automated Recovery


Replica Sets

App Write
Primary
Asynchronous
Read Replication

Secondary
Read

Secondary
Read


Replica Sets

App Write
Primary
Read

Secondary
Read

Secondary
Read


Replica Sets

App
Primary

Write
Primary Automatic Election of
new Primary
Read

Secondary
Read


Replica Sets

App
Recovering

Write New primary serves
Primary data
Read

Secondary
Read


Replica Sets

App
Secondary
Read

Write
Primary
Read

Secondary
Read


Elections

During an election
• Most up to date
• Highest priority
• Less than 10s behind failed Primary


Types of Durability with
MongoDB
• Fire and forget
• Wait for error
• Wait for fsync
• Wait for journal sync
• Wait for replication


Network Ack- Old Default
Driver Primary
write

apply
in
memory


Get last error - New default
Driver Primary
write
getLastError apply
in
memory


Wait for Journal Sync
Driver Primary
write
getLastError apply
in
memory
j:true
Write
to
journal


Wait for replication
Driver Primary Secondary
write
getLastError apply
in
memory
w:2
replicate


Tunable Data Durability
Memory Journal Secondary Other Data Center
RDBMS

network async
ACK
w=1

w=1
j=true sync

w="majority"
w=n
w="myTag"

Less More


Eventual Consistency
Using Replicas for Reads
Read
preference
• primary (only)
• primaryPreferred
• secondary (only)
• secondaryPreferred
• nearest


Immediate Consistency

Thread #1 Primary

Insert v1

Read ✔
Update v2

Read ✔


Eventual Consistency

Thread #1 Primary Secondary Thread #2

Insert v1
v1 does not
exist
Read ✔ ✖
v1
reads v1
Update v2
✔
Read ✔ ✖ reads v1
v2

✔ reads v2


Lab #10
Replication

Primary, Secondary or both?

• Show the latest "votes" for a tweet and/or user
• Changing your proﬁle picture
• Showing your thumbnail with a tweet


Summary

• Schema design is different in MongoDB
• Basic data design principals stay the same
• Focus on how the application manipulates data
• Rapidly evolve schema to meet your requirements
• Consider sharding early
• Understand the impact of eventual consistency


download at mongodb.org

conferences,
appearances,
and
meetups
http://www.10gen.com/events

Facebook

|

Twitter

|

LinkedIn
http://bit.ly/mongo>
@mongodb http://linkd.in/joinmongo


MongoSV Schema Workshop

More Related Content

What's hot

Similar to MongoSV Schema Workshop

More from MongoDB

MongoSV Schema Workshop