BPStudy #42 MongoDB and MyBike.JP and Me

3,569 views
3,524 views

Published on

0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,569
On SlideShare
0
From Embeds
0
Number of Embeds
495
Actions
Shares
0
Downloads
17
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • BPStudy #42 MongoDB and MyBike.JP and Me

    1. 1. 2011/2/22
    2. 2. MongoDBMyBike MongoDB
    3. 3. MongoDB1
    4. 4. Mitsukuni Satoa.k.a. @key3• BeProud• • …• MyBike.JP• MongoDB :Python / Django / PHP /
    5. 5. BSON
    6. 6. BSONMongoDBJSON …
    7. 7. BSONMongoDBJSON …MongoDB shell version: 1.6.6-pre-connecting to: test> foo = { name : Mitsukuni Sato }{ "name" : "Mitsukuni Sato" }> db.people.save(foo)> db.people.find(){ "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato" }
    8. 8. db.collection_name.find( )
    9. 9. db.collection_name.find( )> db.people.find({ name : "Mitsukuni Sato" }){ "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato" }
    10. 10. db.collection_name.save(BSON )
    11. 11. db.collection_name.save(BSON )> user = db.people.findOne({ name : "Mitsukuni Sato" }){ "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato" }> user.age = 3333> db.people.save(user)> user = db.people.findOne({ name : "Mitsukuni Sato" }){ "_id" : ObjectId("4d63d9c48cbc8019697a25de"), "name" : "Mitsukuni Sato", "age" : 33}
    12. 12. GridFS
    13. 13. 1:n master / slave
    14. 14. 1:n master / slave
    15. 15. 1:n master / slave
    16. 16. vs
    17. 17. vsHDD
    18. 18. vsHDD
    19. 19. vsHDD
    20. 20. vsHDD
    21. 21. GridFS 4MB
    22. 22.
    23. 23. source code
    24. 24. source codepre-compiled binary
    25. 25. source codepre-compiled binary .deb (Debian / Ubuntu) RPM (CentOS / Fedora) MacOSX FreeBSD
    26. 26. MySQL
    27. 27. MySQL 1/10
    28. 28. MySQL 1/10
    29. 29. MySQL 1/10
    30. 30. mongodump & mongorestore mongostat mongo
    31. 31. mongodump & mongorestore mongostat mongo
    32. 32. mongodump & mongorestore mongostat mongo
    33. 33. compaction 1GB
    34. 34. ruby, python, C++, Java, Perl, C#
    35. 35. ruby, python, C++, Java, Perl, C#http://www.mongodb.org/display/DOCS/Drivers
    36. 36. ruby, python, C++, Java, Perl, C#http://www.mongodb.org/display/DOCS/Drivers
    37. 37. Python + Django
    38. 38. Python + Django django-mongokit
    39. 39. Python + Django django-mongokit django-mongodb-engine
    40. 40. Python + Django django-mongokit django-mongodb-engine django-storages (GridFS)
    41. 41. MySQL
    42. 42. MySQL
    43. 43. MySQL
    44. 44. MySQL
    45. 45. MySQL django-mongokitModel Document
    46. 46. MySQL + MongoDB
    47. 47. MySQL + MongoDB
    48. 48. MySQL + MongoDB
    49. 49. MySQL + MongoDB metadata object
    50. 50. MySQL + MongoDB metadata objectdjango-mongodb-engine
    51. 51. Document GridFS django-mongokitdjango-mongodb-engine
    52. 52. Document GridFS django-mongokitdjango-mongodb-engine RDB
    53. 53. Document GridFS django-mongokitdjango-mongodb-engine RDB django-mongokit
    54. 54. Document GridFS django-mongokitdjango-mongodb-engine RDB django-mongokit
    55. 55. Document GridFS django-mongokitdjango-mongodb-engine RDB django-mongokit django-mongodb-engine
    56. 56. ManyToMany
    57. 57. ManyToMany tag=A & tag=B
    58. 58. ManyToMany tag=A & tag=B
    59. 59. ManyToMany tag=A & tag=B
    60. 60. ManyToMany tag=A & tag=B
    61. 61. 1.7.5 Write Ahead Log
    62. 62.
    63. 63. 240,000 220,000 200,000 180,000mongod 160,000 140,000 120,000primary Y 100,000 80,000 60,000 40,000 20,000 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 X 240,000 220,000 200,000 180,000 160,000mongod 140,000 bread/s 120,000slave (1) 100,000 80,000 60,000 40,000 20,000 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 X 240,000 220,000 200,000 180,000 160,000mongod 140,000 bread/s 120,000 100,000slave (2) 80,000 60,000 40,000 20,000 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 X
    64. 64. 100 90 80 70 60mongod (%) 50primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800
    65. 65. 100 90 80 70 60mongod (%) 50primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800
    66. 66. 100 90 80 70 60mongod (%) 50primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 nice
    67. 67. 100 90 80 70 60mongod (%) 50primary 40 30 20 10 0 0 45 90 135 180 225 270 315 360 405 450 495 540 585 630 675 720 765 800 nice …
    68. 68. Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specifiedMon Feb 21 06:55:00 [conn25] replSet TODO : dont allow removal of a node until we handle it at theremoved node end?Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2]Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replsetMon Feb 21 06:55:01 [conn25] replSet relinquishing primary stateMon Feb 21 06:55:01 [conn25] replSet RECOVERINGMon Feb 21 06:55:01 [conn25] replSet PRIMARYMon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locallyMon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo",version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" },{ _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036msMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow torespond)Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now upMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY(snip)Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY
    69. 69. Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specifiedMon Feb 21 06:55:00 [conn25] replSet TODO : dont allow removal of a node until we handle it at theremoved node end?Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2]Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replsetMon Feb 21 06:55:01 [conn25] replSet relinquishing primary stateMon Feb 21 06:55:01 [conn25] replSet RECOVERINGMon Feb 21 06:55:01 [conn25] replSet PRIMARYMon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locallyMon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo",version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" },{ _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036msMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow torespond)Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now upMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY(snip)Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY 3GB
    70. 70. Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specifiedMon Feb 21 06:55:00 [conn25] replSet TODO : dont allow removal of a node until we handle it at theremoved node end?Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2]Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replsetMon Feb 21 06:55:01 [conn25] replSet relinquishing primary stateMon Feb 21 06:55:01 [conn25] replSet RECOVERINGMon Feb 21 06:55:01 [conn25] replSet PRIMARYMon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locallyMon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo",version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" },{ _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036msMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow torespond)Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now upMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY(snip)Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY 3GB 6 30
    71. 71. Mon Feb 21 06:55:00 [initandlisten] connection accepted from 127.0.0.1:59385 #25Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig config object parses ok, 3 members specifiedMon Feb 21 06:55:00 [conn25] replSet TODO : dont allow removal of a node until we handle it at theremoved node end?Mon Feb 21 06:55:00 [conn25] replSet replSetReconfig [2]Mon Feb 21 06:55:00 [conn25] replSet info saving a newer config version to local.system.replsetMon Feb 21 06:55:01 [conn25] replSet relinquishing primary stateMon Feb 21 06:55:01 [conn25] replSet RECOVERINGMon Feb 21 06:55:01 [conn25] replSet PRIMARYMon Feb 21 06:55:01 [conn25] replSet replSetReconfig new config saved locallyMon Feb 21 06:55:01 [conn25] query admin.$cmd ntoreturn:1 command: { replSetReconfig: { _id: "foo",version: 2, members: [ { _id: 0, host: "ip-10-118-13-29" }, { _id: 1, host: "ip-10-202-30-58" },{ _id: 2.0, host: "ip-10-195-11-64" } ] } } reslen:53 1036msMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-195-11-64 is now down (or slow torespond)Mon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet info ip-10-118-13-29 is now upMon Feb 21 06:55:01 [ReplSetHealthPollTask] replSet ip-10-118-13-29 SECONDARY(snip)Mon Feb 21 07:01:30 [ReplSetHealthPollTask] replSet ip-10-195-11-64 SECONDARY 3GB 6 30
    72. 72. disk allocation
    73. 73. disk allocation pre allocation
    74. 74. disk allocation pre allocation dd zero padding OK
    75. 75.
    76. 76.
    77. 77.
    78. 78. …20
    79. 79. …20
    80. 80. Amazon EC2 largeMongoDB 1.6.516 connections from workerwrite 12000 qpsselect 100000qps
    81. 81. MyBike.JP
    82. 82. MyBike.JP
    83. 83. MyBike.JP http://mybike.jp
    84. 84. MyBike.JP http://mybike.jp
    85. 85. MyBike.JP http://mybike.jp
    86. 86. MyBike.JP http://mybike.jp
    87. 87. GridFS
    88. 88. GridFS
    89. 89. 7300 objects1GB
    90. 90. Python
    91. 91. Python pymongo
    92. 92. Python pymongo
    93. 93. Python pymongo
    94. 94. Python pymongo …
    95. 95. nginx
    96. 96. nginxnginx-gridfs
    97. 97. nginxnginx-gridfs
    98. 98. GridFS
    99. 99. GridFS django-storages
    100. 100. GridFS django-storages
    101. 101. GridFS django-storages
    102. 102. malloc
    103. 103. malloc32bit
    104. 104. malloc 32bitWed Feb 9 16:52:09 [initandlisten] mmap() failed for /var/lib/mongodb/$tmp_repairDatabase_0/mybike.4 len:536870912 errno:12 Cannot allocate memoryWed Feb 9 16:52:09 [initandlisten] mmap failed with out of memory, ifyoure using 32-bits, then you probably need to upgrade to 64Wed Feb 9 16:52:09 [initandlisten] warning: exception cloning object inmybike.fs.chunks cant map file memory - mongo requires 64 bit build for largerdatasets obj:{ _id: ObjectId(4d44a08c4ae613113700033f), files_id:ObjectId(4d44a08c4ae613113700033e), data: BinData, n: 0 }Wed Feb 9 16:52:09 [initandlisten] mmap failed with out of memory, ifyoure using 32-bits, then you probably need to upgrade to 64
    105. 105. 32bit
    106. 106. 32bit32bit
    107. 107. 32bit32bit2GB
    108. 108. 32bit32bit2GB
    109. 109. 32bit32bit2GB
    110. 110. mongodb stop && mongodb start
    111. 111. mongodb stop && mongodb start
    112. 112. init script
    113. 113. init script10gen Debian
    114. 114. init script10gen Debian …
    115. 115. init script10gen Debian …mongodb stop && mongodb start
    116. 116. init script10gen Debian …mongodb stop && mongodb start stop
    117. 117. init script10gen Debian …mongodb stop && mongodb start stop
    118. 118. Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now
    119. 119. Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now
    120. 120. Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now
    121. 121. Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now
    122. 122. Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now mongodb restart
    123. 123. Mon Feb 21 06:18:04 [interruptThread] shutdown: going to flush oplog...Mon Feb 21 06:18:04 [interruptThread] shutdown: going to close sockets...Mon Feb 21 06:18:04 [interruptThread] shutdown: waiting for fs preallocator...Mon Feb 21 06:18:04 [interruptThread] shutdown: closing all files...Mon Feb 21 06:18:04 [conn1] end connection 127.0.0.1:52525Mon Feb 21 06:18:05 closeAllFiles() finishedMon Feb 21 06:18:05 [interruptThread] shutdown: removing fs lock...Mon Feb 21 06:18:05 dbexit: really exiting now mongodb restart
    124. 124. slave
    125. 125. slave primary
    126. 126. slave primaryslave primary
    127. 127. slave primaryslave primaryprimary / slave
    128. 128. QA

    ×