视觉中国的MongoDB应用实践(QConBeijing2011)
Upcoming SlideShare
Loading in...5
×
 

视觉中国的MongoDB应用实践(QConBeijing2011)

on

  • 9,485 views

在QCon Beijing 2011上做的keynote完整版. 这里包含了因时间限制删除的一些东西,虽然大部分都比较水。

在QCon Beijing 2011上做的keynote完整版. 这里包含了因时间限制删除的一些东西,虽然大部分都比较水。

Statistics

Views

Total Views
9,485
Views on SlideShare
6,022
Embed Views
3,463

Actions

Likes
36
Downloads
352
Comments
6

26 Embeds 3,463

http://nightsailer.com 3045
http://www.cnblogs.com 268
http://www.lifeyun.com 36
http://www.cxybase.com 29
url_unknown 18
http://cache.baidu.com 16
http://buro79xxd.cnblogs.com 10
http://www.dev-club.net 6
http://www.zhuaxia.com 5
http://translate.googleusercontent.com 4
http://www.haogongju.net 4
http://xianguo.com 3
http://cnblogs.com 3
http://reader.aol.com 2
http://cache.baiducontent.com 2
http://www.byywee.com 2
http://archive.cnblogs.com 1
http://localhost 1
http://abtasty.com 1
http://localhost:3000 1
http://unknownerror.net 1
http://webcache.googleusercontent.com 1
http://nightsailer.com HTTP 1
http://translate.google.com 1
http://www.cxforest.com 1
http://www.cnthub.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

15 of 6 Post a comment

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • 收藏了,很好的分享
    Are you sure you want to
    Your message goes here
    Processing…
  • 不错的参考,谢谢分享
    Are you sure you want to
    Your message goes here
    Processing…
  • 非常好的MongoDB经验总结。
    Are you sure you want to
    Your message goes here
    Processing…
  • thnaks a lot
    Are you sure you want to
    Your message goes here
    Processing…
  • 很给力的文档,多谢分享~~~
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

视觉中国的MongoDB应用实践(QConBeijing2011) 视觉中国的MongoDB应用实践(QConBeijing2011) Presentation Transcript

  • QCon beijing 2011 MongoDB: http://czone.chinavisual.com/art/4b4501548f47e8ef73699a0c.html
  • About.me• ( nightsailer) • @nightsailer //twitter,sina,linkedin, github ... • nightsailer # gmail.com • http://nightsailer.com/
  • MongoDB NoSQL ?Auto-shard ?• No! 08 K/V MySQL View slide
  • App MMM • MySQL Percona Mysql(Master-Master) Master-Master-Slaves • HA:MMMMysql(M1) Mysql(M2) Slave Slave View slide
  • •• schema •••
  • MySQL* JSON • schema •* • schema • query
  • App MMM/vdb11 KV KV1 KV2Mysql(M1) Mysql(M2) Slave Slave
  • Memcached• Memcached• KV
  • • PHP/Perl• Memcached•••
  • • Flare• Repcached• Redis• TC/TT
  • Flare• cluster ,• Memcached
  • 1•• • • • ;-(
  • • Cassandra •• CouchDB •
  • MongoDB• Redis• Document• Redis• MySQL ,• MySQL
  • MySQL MongoDB• MySQL MongoDB• MySQL <=> MongoDB•
  • MySQL• Transaction• Joins•
  • ••1 1 90%• 35 table => 10 collection• happy!
  • MongoDB, GridFS•• • MogileFS
  • MongoDB,• SourceForge•• 10gen mailing-list• NoSQL MongoDB
  • 1• comments:{ _id:ObjectId(‘xxx’), art_id:2, content: ‘ ...’, replied_on: 12233 created_on: 12222 replies: [{ _id: ObjectId(‘xxx’), content: ‘ , ...’, replies:[] }]}
  • 2 / :••• / / ...)•
  • 2• db.activity_stream.feed (• db.activity_stream.user )• db.activity_stream(
  • 2feed• • follower collection • embed list
  • 3: 1: Regex• 2: Sphinx• xml•
  • 3: 3: Array/List ($all)•• PHP-SCWS• : • oplog
  • MongoDB• Scons, Python• boost(CentOS)• static link mongod• tcmalloc
  • • MongoDB( ICCCOMMON_CXXFLAGS=-fp-model source -unroll2 -axSSE4.1,SSE4.2 -xSSE3 -static-intel -fpic -fno-strict-aliasingCXXFLAGS="-O3 -ipo -static-libgcc $COMMON_CXXFLAGS"scons --release --static --extrapath=/opt/local --cxx=$CXX --icc --extralib=tcmalloc_minimal --icc-cxxflags="$CXXFLAGS" --icc-cppflags="$CPPFLAGS" -c $BIN_SERVERscons -j4 --release --static --extrapath=/opt/local --cxx=$CXX --icc--extralib=tcmalloc_minimal --icc-cxxflags="$CXXFLAGS" --icc-cppflags="$CPPFLAGS" $BIN_SERVER
  • ••
  • • Raid10• • XFS • Ext4 (?)
  • PHP-FPM 2009/6, 0.9/1.0 • 1 Master + 2 slaves Nginx Mongod Master) Gearman-workers • 1m • 20g Mongod Slave)Gearman-workers • Dell 2850/4g(Master) 2*Dell 2950/4g(
  • * Slave•* lvm snapshot•• fsync & lock db* mongodump•
  • • mongostat / vmstat / iostat• collectd • : json-rest+perl plugin• Munion / Nagios ...
  • MongoDB • CPU • 4G • IO:http://czone.chinavisual.com/art/4c7918b74979590970b80000.html
  • 6:http://czone.chinavisual.com/art/4b45015496ddabef73f49197.html
  • 0:* MongoDB• Out of Memory!• v1.3* cursor• Perl driver bug,
  • 0: 1:•• 2 2:• driver• driver
  • 1:50x 502 Bad Gateway * GridFS • Perl Plack Starman/Plack Nginx Proxy storeMongoDB Disk Cache * GridFS • Nginx proxy_store •
  • 1:50x504 Gateway timeout • • • Perl nginx_errorlog 5 mongodb.log• client
  • 1:50x 3• XFS pre-allocation FS• for i in {1..50} do echo $i head -c 2146435072 /dev/zero > $db.$i done
  • 2:• Mongod crash• Why? • Map/Reduce / •
  • • • 10 • MongoDB • Repair 5 • • 10: http://czone.chinavisual.com/art/4cad0e08497959d621a10000.html
  • • MongoDB • • cluster : slave •
  • 1• --syndelay • 60s(default) => 15~30s • IO• fsync •
  • 2• 1.6.3, Master-Slaves RelicaSets • 1 Primary + 2 Secondary• 4g-8g• w=2 •
  • 5••••
  • :http://czone.chinavisual.com/art/4bd19d3b4979593e1a350000.html
  • 4: RS fail-over• MongoDB • Primary kill 2 secondary • secondary
  • 4: RS fail-over• 2 Arbitor • $ mongod --bind_ip 127.0.0.1,192.168.8.10 -- replSet rs10 --oplogSize 1 ... • > rs.addArb(‘192.168.8.10:27020’) • ...
  • 4: RS fail-over 6: ReplicaSet• 1 Primary + 2 Secondary + n Arbitor••
  • •6 “ ” • ;-( •• ReplicaSet fail-over secondary• Primary , •
  • 1.8• 1.8 • journaling file • mongod -dur • crash repairDatabase•
  • • GridFS • 10mb-500mb •
  • • GridFS • Nginx proxy_store• MongoDB• ,Plack app • prefork
  • • GridFS •• slaveOK• Plack app Twiggy AnyEvent
  • Starman workers/PSGI Starman RS02 Nginx Gearman 127.0.0.1:9001 127.0.0.1:9002 127.0.0.1:9003 127.0.0.1:9004 Twiggy/PSGI RS01 (slaveOK)Sencondary
  • •••
  • Primary • Slave only IDCSecondary1 MongoD Secondary2 ReplicaSet1 • priority 0 VPN • VPN 2 IDC Slave1
  • • >300ms • MongoDB
  • • MongoDB • snapshot• • local.oplog.rs ( tailable cursor) • • replay oplog
  • • 2• GridFS• VPN
  • • VPN• GridFS• • BSON HTTP• mongod
  • 7 • • • • unix: http://czone.chinavisual.com/art/4c05e7be4979596b7e570000.html
  • Auto-sharding• 1.6GA Auto-sharding ••2 1 • •
  • • Shard_key • shard key chunk • 4sq • shard_key• counting • chunk• balancer
  • • • shard • shard•• 1.8
  • •• MongoDB auto-shard• GA • 2.0+ ?
  • MongoDB• MySQL Web• Schema free• Geo• MySQL• GridFS• sharding• Auto-sharding shared_key balancing 1.8/2.0
  • • mongostat • idx missing, faults,global locked • db.serverStatus & rs.status
  • • Replication • oplogsize // slave clone • --fastsync + snapshot • --maxConns • --replSet=<set_name>/<seed list> • seed list 1 arbitor
  • • Some collections • local.oplog.rs/replset.minvalid • $cmd • system.indexes
  • • GridFS • :Write once, Read many. • fs.files _id ETag • file content hash • gc • * Node.js
  • • Update/Delete • ,Schema padding • In-place update • Mark delete remove • • remove() , drop
  • • Group • 4Mb MR• db.eval & Javascript • db • scope •
  • • DBRef •• MR • • MR • v8
  • Question? :http://czone.chinavisual.com/art/4b45015461be3def730e6351.html