MongoDB Sharding - MongoBoston 2010

Scaling and Sharding Eliot Horowitz @eliothorowitz MongoBoston September 20, 2010

Scaling • Data size only goes up • Operations/sec only go up • Vertical scaling is limited • Hard to scale vertically in the cloud • Can scale wider than higher

Traditional Horizontal Scaling • read only slaves • caching • custom partitioning code

Newer Scaling • relational database clustering • consistent hashing (Dynamo) • range based partitioning (BigTable/PNUTS)

MongoDB Sharding • Scale horizontally for data size, index size, write and consistent read scaling • Distribute databases, collections or a objects in a collection • Auto-balancing, migrations, management happen with no down time

• Choose how you partition data • Can convert from single master to sharded system with no downtime • Same features as non-sharding single master • Fully consistent

Range Based • collection is broken into chunks by range • chunks default to 200mb or 100,000 objects

Architecture Shards mongod mongod mongod ... Conﬁg mongod mongod mongod Servers mongod mongod mongod mongos mongos ... client

User proﬁles • Partition by user_id • Secondary indexes on location, dates, etc... • Reads/writes know which shard to hit

User Activity Stream • Shard by user_id • Loading a user’s stream hits a single shard • Writes are distributed across all shards • Can index on activity for deleting

Photos • Can shard by photo_id for best read/write distribution • Secondary index on tags, date

Logging Possible Shard Keys • date • machine, date • logger name

Conﬁg Servers • 3 of them • changes are made with 2 phase commit • if any are down, meta data goes read only • system is online as long as 1/3 is up

Shards • Can be master, master/slave or replica sets • Replica sets gives sharding + full auto- failover • Regular mongod processes

mongos • Sharding Router • Acts just like a mongod to clients • Can have 1 or as many as you want • Can run on appserver so no extra network trafﬁc

Writes • Inserts : require shard key, routed • Removes: routed and/or scattered • Updates: routed or scattered

Queries • By shard key: routed • sorted by shard key: routed in order • by non shard key: scatter gather • sorted by non shard key: distributed merge sort

Operations • split: breaking a chunk into 2 • migrate: move a chunk from 1 shard to another • balancing: moving chunks automatically to keep system in balance

Setting it Up • Start servers • add shards: db.runCommand( { addshard : "10.1.1.5" } ) • turn on partitioning: db.runCommand( { enablesharding : "test" } • shard a collection: db.runCommand( { shardcollection : "test.data" , key : { num : 1 } } )

Download MongoDB http://www.mongodb.org and let us know what you think @eliothorowitz    @mongodb 10gen is hiring! http://www.10gen.com/jobs

http://www.mongodb.org	2265
http://exif.net	112
http://exif.tistory.com	85
http://ru.wiki.mongodb.org	18
http://pt.wiki.mongodb.org	16
http://es.wiki.mongodb.org	11
http://wiki.mongodb.org	4
http://mongodb.onconfluence.com	3
http://www.exif.net	1

MongoDB Sharding - MongoBoston 2010

by Eliot Horowitz on Sep 21, 2010

Accessibility

Tags

Upload Details

Usage Rights

9 Embeds 2,515

Statistics

MongoDB Sharding - MongoBoston 2010 — Presentation Transcript