MongoDB Auto-Sharding at Mongo Seattle
 

MongoDB Auto-Sharding at Mongo Seattle

on

  • 10,544 views

Aaron Staple's presentation at Mongo Seattle

Aaron Staple's presentation at Mongo Seattle

Statistics

Views

Total Views
10,544
Views on SlideShare
10,232
Embed Views
312

Actions

Likes
23
Downloads
301
Comments
1

10 Embeds 312

http://exif.net 123
http://exif.tistory.com 85
http://wiki.simplexi.com 74
http://www.mongodb.org 24
http://wiki.simplexi.com:8080 1
http://mongodb.onconfluence.com 1
http://www.exif.net 1
http://trunk.ly 1
http://slideclip.b-prep.com 1
http://localhost 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Sharding
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

MongoDB Auto-Sharding at Mongo Seattle MongoDB Auto-Sharding at Mongo Seattle Presentation Transcript

  • MongoDBauto sharding
    Aaron Staple
    aaron@10gen.com
    Mongo Seattle
    July 27, 2010
  • MongoDB v1.6out next week!
  • Why Scale Horizontally?
    Vertical scaling is expensive
    Horizontal scaling is more incremental – works well in the cloud
    Will always be able to scale wider than higher
    View slide
  • Distribution Models
    Ad-hoc partitioning
    Consistent hashing (dynamo)
    Range based partitioning (BigTable/PNUTS)
    View slide
  • Auto Sharding
    Each piece of data is exclusively controlled by a single node (shard)
    Each node (shard) has exclusive control over a well defined subset of the data
    Database operations run against one shard when possible, multiple when necessary
    As system load changes, assignment of data to shards is rebalanced automatically
  • Mongo Sharding
    Basic goal is to make this
    mongod
    mongod
    mongod
    ?
    client
  • Mongo Sharding
    Look like this
    mongod
    client
  • Mongo Sharding
    Mapping of documents to shards controlled by shard key
    Can convert from single master to sharded cluster with 0 downtime
    Most functionality of a single Mongo master is preserved
    Fully consistent
  • Shard Key
  • Shard Key Examples
    { user_id: 1 }
    { state: 1 }
    { lastname: 1, firstname: 1 }
    { tag: 1, timestamp: -1 }
    { _id: 1 }
    This is the default
    Careful when using ObjectId
  • Architecture Overview
    5500 <= user_id < +inf
    -inf <= user_id < 2000
    2000 <= user_id < 5500
    mongod
    mongod
    mongod
    mongos
    client
  • Query I
    Shard key { user_id: 1 }
    db.users.find( { user_id: 5000 } )
    Query appropriate shard
  • Query II
    Shard key { user_id: 1 }
    db.users.find( { user_id: { $gt: 4000, $lt: 6000 } } )
    Query appropriate shard(s)
  • { a : …, b : …, c : … } a is declared shard key
    find( { a : { $gt : 333, $lt : 400 } )
  • { a : …, b : …, c : … } a is declared shard key
    find( { a : { $gt : 333, $lt : 2012 } )
  • Query III
    Shard key { user_id: 1 }
    db.users.find( { hometown: ‘Seattle’ } )
    Query all shards
  • { a : …, b : …, c : … } a is declared shard key
    find( { a : { $gt : 333, $lt : 2012 } )
  • { a : …, b : …, c : … } secondary query, secondary index
    ensureIndex({b:1})
    find( { b : 99 } )
    This case good when m is small (such as here), also when the queries are large tasks
  • Query IV
    Shard key { user_id: 1 }
    db.users.find( { hometown: ‘Seattle’ } ).sort( { user_id: 1 } )
    Query all shards, in sequence
  • Query V
    Shard key { user_id: 1 }
    db.users.find( { hometown: ‘Seattle’ } ).sort( { lastname: 1 } )
    Query all shards in parallel, perform merge sort
    Secondary index in { lastname: 1 } can be used
  • Map/Reduce
    Map/Reduce was designed for distributed systems
    Map/Reduce jobs will run on all relevant shards in parallel, subject to query spec
  • Map/Reduce
    > m = function() { emit(this.user_id, 1); }
    > r = function(k,vals) { return 1; }
    > res = db.events.mapReduce(m, r, { query : {type:'sale'} });
    > db[res.result].find().limit(2)
    { "_id" : 8321073716060 , "value" : 1 }
    { "_id" : 7921232311289 , "value" : 1 }
  • Writes
    Inserts routed to appropriate shard (inserted doc must contain shard key)
    Removes call upon matching shards
    Updates call upon matching shards
    Writes parallel if asynchronous, sequential if synchronous
    Updates cannot modify the shard key
  • Choice of Shard Key
    Key per document that is generally a component of your queries
    Often you want a unique key
    If not, consider granularity of key and potentially add fields
    The shard key is generally comprised of fields you would put in an index if operating on a single machine
    But in a sharded configuration, the shard key will be indexed automatically
  • Shard Key Examples (again)
    { user_id: 1 }
    { state: 1 }
    { lastname: 1, firstname: 1 }
    { tag: 1, timestamp: -1 }
    { _id: 1 }
    This is the default
    Careful when using ObjectId
  • Bit.ly Example
    ~50M users
    ~10K concurrently using server at peak
    ~12.5B shortens per month (1K/sec peak)
    History of all shortens per user stored in mongo
  • Bit.ly Example - Schema
    { "_id" : "lthp", "domain" : null, "keyword" : null, "title" : "bit.ly, a simple urlshortener", "url" : "http://jay.bit.ly/", "labels" : [
    {"text" : "lthp",
    "name" : "user_hash"},
    {"text" : "BUFx",
    "name" : "global_hash"},
    {"text" : "http://jay.bit.ly/",
    "name" : "url"},
    {"text" : "bit.ly, a simple urlshortener",
    "name" : "title"}
    ], "ts" : 1229375771, "global_hash" : "BUFx", "user" : "jay", "media_type" : null }
  • Bit.ly Example
    Shard key { user: 1 }
    Indices
    { _id: 1 }
    { user: 1 }
    { global_hash: 1 }
    { user: 1, ts: -1 }
    Query
    db.history.find( { 
    user : user,
    labels.text : …
    } ).sort( { ts: -1 } )
  • Balancing
    The whole point of autosharding is that mongo balances the shards for you
    The balancing algorithm is complicated, basic idea is that this
    1000 <= user_id < +inf
    -inf <= user_id < 1000
  • Balancing
    Becomes this
    3000 <= user_id < +inf
    -inf <= user_id < 3000
  • Balancing
    Current balancing metric is data size
    Future possibilities – cpu, disk utilization
    There is some flexibility built into the partitioning algorithm – so we aren’t thrashing data back and forth between shards
    Only move one ‘chunk’ of data at a time – a conservative choice that limits total overhead of balancing
  • System Architecture
  • Shard
    Regular mongodprocess(es), storing all documents for a given key range
    Handles all reads/writes for this key range as well
    Each shard indexes the data contained within it
    Can be single mongod, master/slave, or replica set
  • Shard - Chunk
    In a sharded cluster, shards partitioned by shard key
    Within a shard, chunks partitioned by shard key
    A chunk is the smallest unit of data for balancing
    Data moves between chunks at chunk granularity
    Upper limit on chunk size is 200MB
    Special case if shard key range is open ended
  • Shard - Replica Sets
    Replica sets provide data redundancy and auto failover
    In the case of sharding, this means redundancy and failover per shard
    All typical replica set operations are possible
    For example, write with w=N
    Replica sets were specifically designed to work as shards
  • System Architecture
  • Mongos
    Sharding router – distributes reads/writes to sharded cluster
    Client interface is the same as a mongod
    Can have as many mongos instances as you want
    Can run on app server machine to avoid extra network traffic
    Mongos also initiates balancing operations
    Keeps metadata per chunk in RAM – 1MB RAM per 1TB of user data in cluster
  • System Architecture
  • Config Server
    3 Config servers
    Changes are made with a 2 phase commit
    If any of the 3 servers goes down, config data becomes read only
    Sharded cluster will remain online as long as 1 of the config servers is running
    Config metadata size estimate
    1MB metadata per 1TB data in cluster
  • Shared Machines
  • Bit.ly Architecture
  • Limitations
    Unique index constraints not expressed by shard key are not enforced across shards
    Updates to a document’s shard key aren’t allowed (you can remove and reinsert, but it’s not atomic)
    Balancing metric is limited to # of chunks right now – but this will be enhanced
    Right now only one chunk moves in the cluster at a time – this means balancing can be slow, it’s a conservative choice we’ve made to keep the overhead of balancing low for now
    20 petabyte size limit
  • Start Up Config Servers
    $ mkdir -p ~/dbs/config
    $ ./mongod --dbpath ~/dbs/config --port 20000
    Repeat as necessary
  • Start Up Mongos
    $ ./mongos --port 30000 --configdb localhost:20000
    No dbpath
    Repeat as necessary
  • Start Up Shards
    $ mkdir -p ~/dbs/shard1
    $ ./mongod --dbpath ~/dbs/shard1 --port 10000
    Repeat as necessary
  • Configure Shards
    $ ./mongo localhost:30000/admin
    > db.runCommand({addshard : "localhost:10000", allowLocal : true})
    {
    "added" : "localhost:10000",
    "ok" : true
    }
    Repeat as necessary
  • Shard Data
    > db.runCommand({"enablesharding" : "foo"})
    > db.runCommand({"shardcollection" : "foo.bar", "key" : {"_id" : 1}})
    Repeat as necessary
  • Production Configuration
    Multiple config servers
    Multiple mongos servers
    Replica sets for each shard
    Use getLastError/w correctly
  • Looking at config data
    Mongo shell connected to config server, config database
    > db.shards.find()
    { "_id" : "shard0", "host" : "localhost:10000" }
    { "_id" : "shard1", "host" : "localhost:10001" }
  • Looking at config data
    > db.databases.find()
    { "_id" : "admin", "partitioned" : false, "primary" : "config" }
    { "_id" : "foo", "partitioned" : false, "primary" : "shard1" }
    { "_id" : "x", "partitioned" : false, "primary" : "shard0” }
    {"_id" : "test", "partitioned" : true, "primary" : "shard0", "sharded" :
    {
    "test.foo" : { "key" : {"x" : 1}, "unique" : false }
    }
    }
  • Looking at config data
    > db.chunks.find()
    {"_id" : "test.foo-x_MinKey",
    "lastmod" : { "t" : 1276636243000, "i" : 1 },
    "ns" : "test.foo",
    "min" : {"x" : { $minKey : 1 } },
    "max" : {"x" : { $maxKey : 1 } },
    "shard" : "shard0”
    }
  • Looking at config data
    > db.printShardingStatus()
    --- Sharding Status ---
    sharding version: { "_id" : 1, "version" : 3 }
    shards:
    { "_id" : "shard0", "host" : "localhost:10000" }
    { "_id" : "shard1", "host" : "localhost:10001" }
    databases:
    { "_id" : "admin", "partitioned" : false, "primary" : "config" }
    { "_id" : "foo", "partitioned" : false, "primary" : "shard1" }
    { "_id" : "x", "partitioned" : false, "primary" : "shard0" }
    { "_id" : "test", "partitioned" : true, "primary" : "shard0","sharded" : { "test.foo" : { "key" : { "x" : 1 }, "unique" : false } } }
    test.foo chunks:
    { "x" : { $minKey : 1 } } -->> { "x" : { $maxKey : 1 } } on : shard0 { "t" : 1276636243 …
  • Give it a Try!
    Download from mongodb.org
    Sharding production ready in 1.6, which is scheduled for release next week
    For now use 1.5 (unstable) to try sharding