Inside MongoDB: the Internals of an Open-Source Database

Mike Dirolf -‐ @mdirolf -‐ 10gen, Inc. http://www.mongodb.org

Inside http://www.ﬂickr.com/photos/tmh9/677919415/

a word of warning this talk might be a bit “hard”, but MongoDB is really easy: http://try.mongodb.org

db.test.insert({hello: “world”})

_id if not speciﬁed drivers will add default: ObjectId("4bface1a2231316e04f3c434") timestamp machine id process id counter http://www.mongodb.org/display/DOCS/Object+IDs

BSON Encoding {_id: ObjectId(XXXXXXXXXXXX), hello: “world”} x27x00x00x00x07 _ i dx00 X X X X X X X X X X X Xx02 h e l l ox00x06x00 x00x00 w o r l dx00x00 http://bsonspec.org

Insert Message (TCP/IP) message length request id response id op code (insert) x68x00x00x00 xXXxXXxXXxXX x00x00x00x00 xd2x07x00x00 reserved collection name document(s) x00x00x00x00 f o o . t e s t x00 BSON Data http://www.mongodb.org/display/DOCS/Mongo+Wire+Protocol

Data File Allocation $ ls -‐sk /data/db/ } 16384 foo.ns allocated per (up to 2 gigs){ double in size 65536 foo.0 131072 foo.1 database 16384 bar.ns ...

Memory Management

Extent Allocation foo.0 allocated per namespace: foo.1 foo.test 00000000000 foo.bar foo.baz 00000000000 00000000000 foo.$freelist 00000000000 0000 preallocated space foo.2 00000000000 00000000000 ns details stored in foo.ns 00000000000 00000000000

Record Allocation ... Header (Size, Oﬀset, Next, Prev) BSON Data Padding Deleted Record (Size, Oﬀset, Next) ...

Indexing B-‐Tree indexes, stored in own namespaces > db.system.namespaces.find() { "name" : "foo.system.indexes" } { "name" : "foo.test" } { "name" : "foo.test.$_id_" } http://www.mongodb.org/display/DOCS/Indexes

db.test.find({hello: “world”})

Query Language “query by example” plus $ modiﬁers: {first_name: “Mike”, age: {$gte: 20, $lt: 40}} http://www.mongodb.org/display/DOCS/Advanced+Queries

Cursors > var c = db.test.find({x: 20}).skip(20).limit(10) > c.next() > c.next() ... query ﬁrst N results + cursor id getMore w/ cursor id next N results + cursor id or 0 ...

Query Optimizer find({x: 10, y: “foo”}) scan terminate index on x index on y remember

db.foo.drop()

Commands drop, count, copydb, ﬁndAndModify, serverStatus, ... http://www.mongodb.org/display/DOCS/Commands

db.foo.drop(); = db.foo.runCommand({drop: "foo"}); = db.$cmd.findOne({drop: "foo"}); = db.$cmd.find({drop: "foo"}).limit(-‐1);

Capped Collections preallocated auto LRI age-‐out no default _id index always in insertion order http://www.mongodb.org/display/DOCS/Capped+Collections

Replication Oplog > use local switched to db local > use foo > db.oplog.$main.find() switched to db foo {ts: ..., op: "n", ns: "", o: {}} {ts: ..., op: "n", ns: "", o: {}} > db.test.insert({x: 1, url: {ts: ..., op: "i", ns: "foo.test", "http://dirolf.com"}); o: {_id: ObjectId("..."), x: 1, url: "http://dirolf.com"}} {ts: ..., op: "n", ns: "", o: {}} > db.test.update({url: "http:// {ts: ..., op: "u", ns: "foo.test", dirolf.com"}, {$inc: {x: 1}}); o2: {_id: ObjectId("...")}, o: {$set: {x: 2}}} http://www.mongodb.org/display/DOCS/Replication

Replication Topology master slave master master slave slave slave slave master master slave master http://www.mongodb.org/display/DOCS/Replication

Auto-Sharding Shards mongod mongod mongod ... Conﬁg mongod mongod mongod Servers mongod mongod mongod mongos mongos ... client http://www.mongodb.org/display/DOCS/Sharding

Geohashing (20, 10) (0001 0100, 0000 1010) 0000 0010 0110 0100 maps close coordinates (21, 9) to close hashes: 0000 0010 0110 0011 tricky part happens at bit-‐ﬂips (127 vs 128) http://www.mongodb.org/display/DOCS/Geospatial+Indexing

Download MongoDB http://www.mongodb.org these slides are available at http://dirolf.com

http://blog.nosqlfan.com	1223
http://www.webguo.com	949
http://nosql.mypopescu.com	670
http://dirolf.com	637
http://www.slideshare.net	142
http://blog.csdn.net	57
http://202.108.48.24	28
http://www.w3ccollege.org	20
http://static.slidesharecdn.com	8
http://www.mongodb.org	8
http://cache.baidu.com	6
http://trunk.ly	5
http://www.oschina.net	4
http://webguoblog.appspot.com	4
http://xianguo.com	2
http://www.techgig.com	2
http://facebook.slideshare.com	2
http://translate.googleusercontent.com	2
http://de52ry.wordpress.com	1
http://my.oschina.net	1
http://ctc.qzs.qq.com	1
http://reader.youdao.com	1
http://paper.li	1
http://webcache.googleusercontent.com	1
http://localhost:4000	1

Inside MongoDB: the Internals of an Open-Source Database

by Mike Dirolf on May 27, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

25 Embeds 3,776

Statistics

Inside MongoDB: the Internals of an Open-Source Database — Presentation Transcript