BPStudy #42  MongoDB and MyBike.JP and Me
Upcoming SlideShare
Loading in...5
×
 

BPStudy #42 MongoDB and MyBike.JP and Me

on

  • 3,667 views

 

Statistics

Views

Total Views
3,667
Views on SlideShare
3,205
Embed Views
462

Actions

Likes
5
Downloads
16
Comments
0

3 Embeds 462

http://blog.mitsukuni.org 454
http://webcache.googleusercontent.com 5
http://feeds.feedburner.com 3

Accessibility

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

BPStudy #42  MongoDB and MyBike.JP and Me BPStudy #42 MongoDB and MyBike.JP and Me Presentation Transcript

  • 2011/2/22
  • MongoDBMyBike MongoDB
  • MongoDB1
  • Mitsukuni Satoa.k.a. @key3• BeProud• • …• MyBike.JP• MongoDB :Python / Django / PHP /
  • BSON
  • BSONMongoDBJSON …
  • BSONMongoDBJSON …MongoDB shell version: 1.6.6-pre-connecting to: test> foo = { name : Mitsukuni Sato }{ "name" : "Mitsukuni Sato" }> db.people.save(foo)> db.people.find(){ "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato" }
  • db.collection_name.find( )
  • db.collection_name.find( )> db.people.find({ name : "Mitsukuni Sato" }){ "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato" }
  • db.collection_name.save(BSON )
  • db.collection_name.save(BSON )> user = db.people.findOne({ name : "Mitsukuni Sato" }){ "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato" }> user.age = 3333> db.people.save(user)> user = db.people.findOne({ name : "Mitsukuni Sato" }){ "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato", "age" : 33}
  • GridFS
  • 1:n master / slave
  • 1:n master / slave
  • 1:n master / slave
  • vs
  • vsHDD
  • vsHDD
  • vsHDD
  • vsHDD
  • GridFS 4MB
  • source code
  • source codepre-compiled binary
  • source codepre-compiled binary .deb (Debian / Ubuntu) RPM (CentOS / Fedora) MacOSX FreeBSD
  • MySQL
  • MySQL 1/10
  • MySQL 1/10
  • MySQL 1/10
  • mongodump & mongorestore mongostat mongo
  • mongodump & mongorestore mongostat mongo
  • mongodump & mongorestore mongostat mongo
  • compaction 1GB
  • ruby, python, C++, Java, Perl, C#
  • ruby, python, C++, Java, Perl, C#http://www.mongodb.org/display/DOCS/Drivers
  • ruby, python, C++, Java, Perl, C#http://www.mongodb.org/display/DOCS/Drivers
  • Python + Django
  • Python + Django django-mongokit
  • Python + Django django-mongokit django-mongodb-engine
  • Python + Django django-mongokit django-mongodb-engine django-storages (GridFS)
  • MySQL
  • MySQL
  • MySQL
  • MySQL
  • MySQL django-mongokitModel Document
  • MySQL + MongoDB
  • MySQL + MongoDB
  • MySQL + MongoDB
  • MySQL + MongoDB metadata object
  • MySQL + MongoDB metadata objectdjango-mongodb-engine
  • Document GridFS django-mongokitdjango-mongodb-engine
  • Document GridFS django-mongokitdjango-mongodb-engine RDB
  • Document GridFS django-mongokitdjango-mongodb-engine RDB django-mongokit
  • Document GridFS django-mongokitdjango-mongodb-engine RDB django-mongokit
  • Document GridFS django-mongokitdjango-mongodb-engine RDB django-mongokit django-mongodb-engine
  • ManyToMany
  • ManyToMany tag=A & tag=B
  • ManyToMany tag=A & tag=B
  • ManyToMany tag=A & tag=B
  • ManyToMany tag=A & tag=B
  • 1.7.5 Write Ahead Log
  • 240,000 220,000 200,000 180,000mongod 160,000 140,000 120,000primary Y 100,000 80,000 60,000 40,000 20,000 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 X 240,000 220,000 200,000 180,000 160,000mongod 140,000 bread/s 120,000slave (1) 100,000 80,000 60,000 40,000 20,000 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 X 240,000 220,000 200,000 180,000 160,000mongod 140,000 bread/s 120,000 100,000slave (2) 80,000 60,000 40,000 20,000 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 X
  • 100 90 80 70 60mongod (%) 50primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800
  • 100 90 80 70 60mongod (%) 50primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800
  • 100 90 80 70 60mongod (%) 50primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 nice
  • 100 90 80 70 60mongod (%) 50primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 nice …
  • Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specifiedMon Feb 21 06:55:00 [conn25] replSet TODO : dont allow removal of a node until we handle it at theremoved node end?Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2]Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replsetMon Feb 21 06:55:01 [conn25] replSet relinquishing primary stateMon Feb 21 06:55:01 [conn25] replSet RECOVERINGMon Feb 21 06:55:01 [conn25] replSet PRIMARYMon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locallyMon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo",version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" },{ _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036msMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow torespond)Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now upMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY(snip)Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY
  • Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specifiedMon Feb 21 06:55:00 [conn25] replSet TODO : dont allow removal of a node until we handle it at theremoved node end?Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2]Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replsetMon Feb 21 06:55:01 [conn25] replSet relinquishing primary stateMon Feb 21 06:55:01 [conn25] replSet RECOVERINGMon Feb 21 06:55:01 [conn25] replSet PRIMARYMon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locallyMon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo",version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" },{ _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036msMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow torespond)Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now upMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY(snip)Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY 3GB
  • Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specifiedMon Feb 21 06:55:00 [conn25] replSet TODO : dont allow removal of a node until we handle it at theremoved node end?Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2]Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replsetMon Feb 21 06:55:01 [conn25] replSet relinquishing primary stateMon Feb 21 06:55:01 [conn25] replSet RECOVERINGMon Feb 21 06:55:01 [conn25] replSet PRIMARYMon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locallyMon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo",version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" },{ _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036msMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow torespond)Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now upMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY(snip)Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY 3GB 6 30
  • Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specifiedMon Feb 21 06:55:00 [conn25] replSet TODO : dont allow removal of a node until we handle it at theremoved node end?Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2]Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replsetMon Feb 21 06:55:01 [conn25] replSet relinquishing primary stateMon Feb 21 06:55:01 [conn25] replSet RECOVERINGMon Feb 21 06:55:01 [conn25] replSet PRIMARYMon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locallyMon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo",version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" },{ _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036msMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow torespond)Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now upMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY(snip)Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY 3GB 6 30
  • disk allocation
  • disk allocation pre allocation
  • disk allocation pre allocation dd zero padding OK
  • …20
  • …20
  • Amazon EC2 largeMongoDB 1.6.516 connections from workerwrite 12000 qpsselect 100000qps
  • MyBike.JP
  • MyBike.JP
  • MyBike.JP http://mybike.jp
  • MyBike.JP http://mybike.jp
  • MyBike.JP http://mybike.jp
  • MyBike.JP http://mybike.jp
  • GridFS
  • GridFS
  • 7300 objects1GB
  • Python
  • Python pymongo
  • Python pymongo
  • Python pymongo
  • Python pymongo …
  • nginx
  • nginxnginx-gridfs
  • nginxnginx-gridfs
  • GridFS
  • GridFS django-storages
  • GridFS django-storages
  • GridFS django-storages
  • malloc
  • malloc32bit
  • malloc 32bitWed Feb 9 16:52:09 [initandlisten] mmap() failed for /var/lib/mongodb/$tmp_repairDatabase_0/mybike.4 len:536870912 errno:12 Cannot allocate memoryWed Feb 9 16:52:09 [initandlisten] mmap failed with out of memory, ifyoure using 32-bits, then you probably need to upgrade to 64Wed Feb 9 16:52:09 [initandlisten] warning: exception cloning object inmybike.fs.chunks cant map file memory - mongo requires 64 bit build for largerdatasets obj:{ _id: ObjectId(4d44a08c4ae613113700033f), files_id:ObjectId(4d44a08c4ae613113700033e), data: BinData, n: 0 }Wed Feb 9 16:52:09 [initandlisten] mmap failed with out of memory, ifyoure using 32-bits, then you probably need to upgrade to 64
  • 32bit
  • 32bit32bit
  • 32bit32bit2GB
  • 32bit32bit2GB
  • 32bit32bit2GB
  • mongodb stop && mongodb start
  • mongodb stop && mongodb start
  • init script
  • init script10gen Debian
  • init script10gen Debian …
  • init script10gen Debian …mongodb stop && mongodb start
  • init script10gen Debian …mongodb stop && mongodb start stop
  • init script10gen Debian …mongodb stop && mongodb start stop
  • Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now
  • Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now
  • Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now
  • Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now
  • Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now mongodb restart
  • Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now mongodb restart
  • slave
  • slave primary
  • slave primaryslave primary
  • slave primaryslave primaryprimary / slave
  • QA