QCon beijing 2011                                                                   MongoDB: http://czone.chinavisual.com/...
About.me•         ( nightsailer)    • @nightsailer //twitter,sina,linkedin,      github ...    • nightsailer # gmail.com  ...
MongoDB  NoSQL      ?Auto-shard           ?• No! 08                           K/V    MySQL
App               MMM                                                                  • MySQL   Percona                  ...
••       schema    •••
MySQL*                JSON    • schema    •*    • schema    •          query
App            MMM/vdb11                                                        KV                                        ...
Memcached•     Memcached•                KV
•   PHP/Perl•   Memcached•••
• Flare• Repcached• Redis• TC/TT
Flare•      cluster     ,•      Memcached
1••    •    •    •       ;-(
• Cassandra •• CouchDB •
MongoDB•                Redis• Document•            Redis• MySQL              ,• MySQL
MySQL   MongoDB•          MySQL     MongoDB•        MySQL <=> MongoDB•
MySQL• Transaction• Joins•
••1   1            90%•     35   table => 10 collection•            happy!
MongoDB,           GridFS••    •   MogileFS
MongoDB,• SourceForge•• 10gen         mailing-list• NoSQL                MongoDB
1•    comments:{      _id:ObjectId(‘xxx’),      art_id:2,      content: ‘           ...’,      replied_on:    12233      c...
2           /       :•••               /       /    ...)•
2• db.activity_stream.feed (• db.activity_stream.user         )• db.activity_stream(
2feed•    •                follower        collection    •                  embed list
3:    1: Regex•    2: Sphinx•              xml•
3:        3:        Array/List      ($all)••            PHP-SCWS•             :    •                          oplog
MongoDB• Scons,         Python• boost(CentOS)• static link mongod• tcmalloc
•                MongoDB( ICCCOMMON_CXXFLAGS=-fp-model source -unroll2   -axSSE4.1,SSE4.2 -xSSE3 -static-intel -fpic -fno-...
••
•            Raid10•    • XFS    • Ext4 (?)
PHP-FPM                                          2009/6,       0.9/1.0                                    • 1 Master + 2 s...
*          Slave•* lvm snapshot•• fsync & lock db* mongodump•
• mongostat / vmstat / iostat• collectd  •        : json-rest+perl plugin•         Munion / Nagios ...
MongoDB                              • CPU                              •                                        4G       ...
6:http://czone.chinavisual.com/art/4b45015496ddabef73f49197.html
0:* MongoDB• Out of Memory!•        v1.3*       cursor• Perl driver    bug,
0:    1:••                          2    2:•              driver•                 driver
1:50x                                             502 Bad Gateway                                             * GridFS    ...
1:50x504 Gateway timeout    •    •    • Perl            nginx_errorlog             5          mongodb.log•                ...
1:50x    3•               XFS                      pre-allocation FS•        for i in {1..50}        do            echo $i...
2:• Mongod crash•                Why?  •               Map/Reduce   / •
•                      • 10                      • MongoDB                       • Repair                                 ...
• MongoDB • • cluster   :   slave •
1•                      --syndelay    • 60s(default) => 15~30s    • IO•                          fsync    •
2•          1.6.3,     Master-Slaves RelicaSets    • 1 Primary + 2 Secondary•                   4g-8g•                    ...
5••••
:http://czone.chinavisual.com/art/4bd19d3b4979593e1a350000.html
4: RS fail-over• MongoDB • Primary             kill   2     secondary •         secondary
4: RS fail-over•          2   Arbitor    •   $ mongod --bind_ip 127.0.0.1,192.168.8.10 --        replSet rs10 --oplogSize ...
4: RS fail-over    6:             ReplicaSet• 1 Primary + 2 Secondary + n Arbitor••
•6          “ ”  •                     ;-(  ••          ReplicaSet   fail-over  secondary• Primary          , •
1.8• 1.8  • journaling file  • mongod -dur  • crash            repairDatabase•
• GridFS •           10mb-500mb •
• GridFS  •           Nginx proxy_store• MongoDB•               ,Plack app  • prefork
• GridFS  ••           slaveOK• Plack app       Twiggy   AnyEvent
Starman workers/PSGI                              Starman             RS02                                                ...
•••
Primary                                           •                        Slave only                               IDCSec...
•               >300ms    • MongoDB
•       MongoDB    •   snapshot•    •     local.oplog.rs (   tailable cursor)    •    •               replay oplog
•        2• GridFS•     VPN
•              VPN•     GridFS•    • BSON             HTTP•                    mongod
7                                                                   •                                                     ...
Auto-sharding• 1.6GA   Auto-sharding ••2        1 • •
• Shard_key • shard key        chunk   • 4sq • shard_key• counting •          chunk• balancer
•    •         shard    •             shard••       1.8
•• MongoDB    auto-shard• GA  • 2.0+ ?
MongoDB• MySQL Web• Schema free• Geo• MySQL• GridFS•                            sharding• Auto-sharding shared_key   balan...
• mongostat • idx missing, faults,global locked • db.serverStatus & rs.status
• Replication •            oplogsize //    slave     clone  • --fastsync + snapshot  • --maxConns  • --replSet=<set_name>/...
• Some collections   • local.oplog.rs/replset.minvalid   • $cmd   • system.indexes
• GridFS •           :Write once, Read many. • fs.files _id ETag • file content hash • gc • * Node.js
• Update/Delete •           ,Schema          padding   • In-place update • Mark delete            remove   • • remove() , ...
• Group • 4Mb              MR• db.eval & Javascript •                           db •                       scope •
• DBRef •• MR • •          MR   • v8
Question?      :http://czone.chinavisual.com/art/4b45015461be3def730e6351.html
Upcoming SlideShare
Loading in...5
×

视觉中国的MongoDB应用实践(QConBeijing2011)

9,590

Published on

在QCon Beijing 2011上做的keynote完整版. 这里包含了因时间限制删除的一些东西,虽然大部分都比较水。

Published in: Technology
6 Comments
36 Likes
Statistics
Notes
  • 收藏了,很好的分享
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • 不错的参考,谢谢分享
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • 非常好的MongoDB经验总结。
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • thnaks a lot
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • 很给力的文档,多谢分享~~~
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
9,590
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
359
Comments
6
Likes
36
Embeds 0
No embeds

No notes for slide

视觉中国的MongoDB应用实践(QConBeijing2011)

  1. 1. QCon beijing 2011 MongoDB: http://czone.chinavisual.com/art/4b4501548f47e8ef73699a0c.html
  2. 2. About.me• ( nightsailer) • @nightsailer //twitter,sina,linkedin, github ... • nightsailer # gmail.com • http://nightsailer.com/
  3. 3. MongoDB NoSQL ?Auto-shard ?• No! 08 K/V MySQL
  4. 4. App MMM • MySQL Percona Mysql(Master-Master) Master-Master-Slaves • HA:MMMMysql(M1) Mysql(M2) Slave Slave
  5. 5. •• schema •••
  6. 6. MySQL* JSON • schema •* • schema • query
  7. 7. App MMM/vdb11 KV KV1 KV2Mysql(M1) Mysql(M2) Slave Slave
  8. 8. Memcached• Memcached• KV
  9. 9. • PHP/Perl• Memcached•••
  10. 10. • Flare• Repcached• Redis• TC/TT
  11. 11. Flare• cluster ,• Memcached
  12. 12. 1•• • • • ;-(
  13. 13. • Cassandra •• CouchDB •
  14. 14. MongoDB• Redis• Document• Redis• MySQL ,• MySQL
  15. 15. MySQL MongoDB• MySQL MongoDB• MySQL <=> MongoDB•
  16. 16. MySQL• Transaction• Joins•
  17. 17. ••1 1 90%• 35 table => 10 collection• happy!
  18. 18. MongoDB, GridFS•• • MogileFS
  19. 19. MongoDB,• SourceForge•• 10gen mailing-list• NoSQL MongoDB
  20. 20. 1• comments:{ _id:ObjectId(‘xxx’), art_id:2, content: ‘ ...’, replied_on: 12233 created_on: 12222 replies: [{ _id: ObjectId(‘xxx’), content: ‘ , ...’, replies:[] }]}
  21. 21. 2 / :••• / / ...)•
  22. 22. 2• db.activity_stream.feed (• db.activity_stream.user )• db.activity_stream(
  23. 23. 2feed• • follower collection • embed list
  24. 24. 3: 1: Regex• 2: Sphinx• xml•
  25. 25. 3: 3: Array/List ($all)•• PHP-SCWS• : • oplog
  26. 26. MongoDB• Scons, Python• boost(CentOS)• static link mongod• tcmalloc
  27. 27. • MongoDB( ICCCOMMON_CXXFLAGS=-fp-model source -unroll2 -axSSE4.1,SSE4.2 -xSSE3 -static-intel -fpic -fno-strict-aliasingCXXFLAGS="-O3 -ipo -static-libgcc $COMMON_CXXFLAGS"scons --release --static --extrapath=/opt/local --cxx=$CXX --icc --extralib=tcmalloc_minimal --icc-cxxflags="$CXXFLAGS" --icc-cppflags="$CPPFLAGS" -c $BIN_SERVERscons -j4 --release --static --extrapath=/opt/local --cxx=$CXX --icc--extralib=tcmalloc_minimal --icc-cxxflags="$CXXFLAGS" --icc-cppflags="$CPPFLAGS" $BIN_SERVER
  28. 28. ••
  29. 29. • Raid10• • XFS • Ext4 (?)
  30. 30. PHP-FPM 2009/6, 0.9/1.0 • 1 Master + 2 slaves Nginx Mongod Master) Gearman-workers • 1m • 20g Mongod Slave)Gearman-workers • Dell 2850/4g(Master) 2*Dell 2950/4g(
  31. 31. * Slave•* lvm snapshot•• fsync & lock db* mongodump•
  32. 32. • mongostat / vmstat / iostat• collectd • : json-rest+perl plugin• Munion / Nagios ...
  33. 33. MongoDB • CPU • 4G • IO:http://czone.chinavisual.com/art/4c7918b74979590970b80000.html
  34. 34. 6:http://czone.chinavisual.com/art/4b45015496ddabef73f49197.html
  35. 35. 0:* MongoDB• Out of Memory!• v1.3* cursor• Perl driver bug,
  36. 36. 0: 1:•• 2 2:• driver• driver
  37. 37. 1:50x 502 Bad Gateway * GridFS • Perl Plack Starman/Plack Nginx Proxy storeMongoDB Disk Cache * GridFS • Nginx proxy_store •
  38. 38. 1:50x504 Gateway timeout • • • Perl nginx_errorlog 5 mongodb.log• client
  39. 39. 1:50x 3• XFS pre-allocation FS• for i in {1..50} do echo $i head -c 2146435072 /dev/zero > $db.$i done
  40. 40. 2:• Mongod crash• Why? • Map/Reduce / •
  41. 41. • • 10 • MongoDB • Repair 5 • • 10: http://czone.chinavisual.com/art/4cad0e08497959d621a10000.html
  42. 42. • MongoDB • • cluster : slave •
  43. 43. 1• --syndelay • 60s(default) => 15~30s • IO• fsync •
  44. 44. 2• 1.6.3, Master-Slaves RelicaSets • 1 Primary + 2 Secondary• 4g-8g• w=2 •
  45. 45. 5••••
  46. 46. :http://czone.chinavisual.com/art/4bd19d3b4979593e1a350000.html
  47. 47. 4: RS fail-over• MongoDB • Primary kill 2 secondary • secondary
  48. 48. 4: RS fail-over• 2 Arbitor • $ mongod --bind_ip 127.0.0.1,192.168.8.10 -- replSet rs10 --oplogSize 1 ... • > rs.addArb(‘192.168.8.10:27020’) • ...
  49. 49. 4: RS fail-over 6: ReplicaSet• 1 Primary + 2 Secondary + n Arbitor••
  50. 50. •6 “ ” • ;-( •• ReplicaSet fail-over secondary• Primary , •
  51. 51. 1.8• 1.8 • journaling file • mongod -dur • crash repairDatabase•
  52. 52. • GridFS • 10mb-500mb •
  53. 53. • GridFS • Nginx proxy_store• MongoDB• ,Plack app • prefork
  54. 54. • GridFS •• slaveOK• Plack app Twiggy AnyEvent
  55. 55. Starman workers/PSGI Starman RS02 Nginx Gearman 127.0.0.1:9001 127.0.0.1:9002 127.0.0.1:9003 127.0.0.1:9004 Twiggy/PSGI RS01 (slaveOK)Sencondary
  56. 56. •••
  57. 57. Primary • Slave only IDCSecondary1 MongoD Secondary2 ReplicaSet1 • priority 0 VPN • VPN 2 IDC Slave1
  58. 58. • >300ms • MongoDB
  59. 59. • MongoDB • snapshot• • local.oplog.rs ( tailable cursor) • • replay oplog
  60. 60. • 2• GridFS• VPN
  61. 61. • VPN• GridFS• • BSON HTTP• mongod
  62. 62. 7 • • • • unix: http://czone.chinavisual.com/art/4c05e7be4979596b7e570000.html
  63. 63. Auto-sharding• 1.6GA Auto-sharding ••2 1 • •
  64. 64. • Shard_key • shard key chunk • 4sq • shard_key• counting • chunk• balancer
  65. 65. • • shard • shard•• 1.8
  66. 66. •• MongoDB auto-shard• GA • 2.0+ ?
  67. 67. MongoDB• MySQL Web• Schema free• Geo• MySQL• GridFS• sharding• Auto-sharding shared_key balancing 1.8/2.0
  68. 68. • mongostat • idx missing, faults,global locked • db.serverStatus & rs.status
  69. 69. • Replication • oplogsize // slave clone • --fastsync + snapshot • --maxConns • --replSet=<set_name>/<seed list> • seed list 1 arbitor
  70. 70. • Some collections • local.oplog.rs/replset.minvalid • $cmd • system.indexes
  71. 71. • GridFS • :Write once, Read many. • fs.files _id ETag • file content hash • gc • * Node.js
  72. 72. • Update/Delete • ,Schema padding • In-place update • Mark delete remove • • remove() , drop
  73. 73. • Group • 4Mb MR• db.eval & Javascript • db • scope •
  74. 74. • DBRef •• MR • • MR • v8
  75. 75. Question? :http://czone.chinavisual.com/art/4b45015461be3def730e6351.html
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×