Download as PDF, PPTX
![Developing Application
• Define data without using transaction
- User data are defined in 1 document
and updated only once in block
User Data
{ facebookId : xxx,
Image
status : { lv : 10, coin : 9999, ... },
layerInfo : 1¦1¦0¦1¦2¦1¦1¦3¦1¦1¦4¦0... ,
structure : {
1¦1 : { id : xxxx, depth : 3, width : 3, aspect : 1, ... }, ...
},
inventory : [ { id : xxx, size : xxx, ... }, ... ],
neighbor : [ { id : xxx, newFlg : false, ... }, ... ],
animal : [ { id : xxx, color : 0, pos : 20¦20 , ... } ],
...
}](https://image.slidesharecdn.com/mongotokyo2012en-120119042747-phpapp02/85/The-Case-for-using-MongoDB-in-Social-Game-Animal-Land-20-320.jpg)
Uploaded byMasakazu Matsushita
The Case for using MongoDB in Social Game - Animal Land
The document discusses the implementation of MongoDB in the social game 'Animal Land' developed by CyberAgent, Inc. It outlines the system architecture, benefits of MongoDB such as scalability and flexibility with data structures, and various development considerations including performance, infrastructure, and troubleshooting. Current challenges include online backups and shard management during upgrades.
More Related Content
Similar to The Case for using MongoDB in Social Game - Animal Land
The Case for using MongoDB in Social Game - Animal Land
- 1. The Case forusing MongoDB in Social Game Masakazu Matsushita Cyberagent, Inc.
- 2. About Me • MasakazuMatsushita • @matsukaz • Cyberagent, Inc. - SocialApps Division - Ameba Pico (2011/01∼) - Animal Land (2011/03∼) • DevLOVE Staff
- 3. Agenda • About AnimalLand • System Overview • Why MongoDB ? • Considered Points • Troubles • Current Problems
- 4.
- 5.
- 6. Development Period • Startedon 2011/03 • First meeting was held on 3.11 • Released on 2011/12/09
- 7. Team Member • Producer2 • Designer 1 • Flash Developer 3 • Engineer 4 +α
- 8.
- 9. Using Cloud Services •Amazon Web Services - EC2 (Instance Store + EBS) - S3 - CloudFront - Route53 - Elastic Load Balancing • Google Apps • Kontagent
- 10. System Relations iframe API Call Callback for Payment API Web HTML Server JSON Flash Command AMF Server AMF : Actionscript Message Format
- 11. L m1.large Servers XX EBS m2.2xlarge EBS ELB ELB S3 Web L 3 Command L 4 CloudFront nginx nginx Route 53 Tomcat Tomcat バッチ L mongos mongos バッチ MySQL Shard 5 MySQL L memcached L 2 monitor L MongoDB XX MySQL memcached EBS munin mongod MySQL L nagios MongoDB XX MySQL admin L ビルド L mongod EBS nginx redmine MongoDB XX MongoDB XX 3 Tomcat jenkins mongod mongoc EBS EBS mongos Maven SVN EBS Secondary
- 12. Middleware • nginx 1.0.x •Tomcat 7.0.x • MongoDB 2.0.1 • MySQL 5.5.x • memcached 1.4.x
- 13. Framework / Libraries •Spring Framework、Spring MVC • BlazeDS、Spring Flex • Spring Data - MongoDB 1.0.0 M4 • mongo-java-driver 2.6.5 • Ehcache • spymemcached • RestFB • MyBatis
- 14.
- 15. • Used inAmeba Pico http://apps.facebook.com/amebapico/
- 16. • I likeMongoDB features !! - ReplicaSet - Auto-Sharding - Can handle complex data structures - Index, Advanced Queries - Developed actively - Quite easy to understand
- 17. • Fits inAnimal Land requirements - Complex data structures (City s grid data, user/structure parameters, ...) - Most processes run sequencially and doesn t update same data at a time - Maintenance-less • Change data structures dynamically • Durability, Scalability
- 18. • Resolve MongoDBs weak points by other way - Some data are stored in other DB • Payments history needs reliability, so they are stored in MySQL • Temporary data are stored in memcached - Don t think of using transactions
- 19.
- 20. Developing Application • Definedata without using transaction - User data are defined in 1 document and updated only once in block User Data { facebookId : xxx, Image status : { lv : 10, coin : 9999, ... }, layerInfo : 1¦1¦0¦1¦2¦1¦1¦3¦1¦1¦4¦0... , structure : { 1¦1 : { id : xxxx, depth : 3, width : 3, aspect : 1, ... }, ... }, inventory : [ { id : xxx, size : xxx, ... }, ... ], neighbor : [ { id : xxx, newFlg : false, ... }, ... ], animal : [ { id : xxx, color : 0, pos : 20¦20 , ... } ], ... }
- 21. Developing Application • Cutdowndata size as possible - City s grid data are stored in 1 field (Expanded when application uses) - Data in design phase（500 KB） layerInfo : { 1¦1 : 0, 1¦2 : 1, .... } - Current data（50 KB） layerInfo : 1¦1¦0¦1¦2¦1¦1¦3¦1¦1¦4¦0...
- 22. Developing Application • Carefulnot to define too many fields - It took more than 5 sec for find() when data contained more than 20,000 fields (144x144 City s grid data) - Consider data size, and also for the time to parse BSON
- 23. Developing Application • ShardKey is decided in following policy - Don t use low-cardinality key - Recent used data should be on memory, and data not used should not be - Frequently read/write data should be in each shard in nice balance
- 24. Developing Application • Usetargeted operation as possible - Use Shard Key - Use index for non Shard Key op Operation Type db.foo.find({ShardKey : 1}) Targeted db.foo.find({ShardKey : 1, NonShardKey : 1}) Targeted db.foo.find({NonShardKey : 1}) Global db.foo.find() Global db.foo.insert(<object>) Targeted db.foo.update({ShardKey : 1}, <object>) Targeted db.foo.remove({ShardKey : 1}) db.foo.update({NonShardKey : 1}, <object>) Global db.foo.remove({NonShardKey : 1})
- 25. Developing Application • Decreaseupdate frequency - Store master data in memory - Queue mechanism in Flash - User operations are processed in block (once per 5 sec) • Server processes sequencially Processes User op. Stored in Sequencially Queue Queue Command Flash Server Send in Block once per 5 sec
- 26. Developing Application • Developefficiently using O/R Mapper - Spring Data - MongoDB and Wrapper classes @Autowired protected MongoTemplate mongoTemplate; public void insert(T entity) { mongoTemplate.save(entity); } - Can use any serializable object - Maybe easier to deal with than RDB O/R Mapper
- 27. Developing Application • Implementswithout expecting rollback - Give up for some inconsistent data - Careful not to inflict disadvantages to user
- 28. Constructing Infrastructure • Estimatesame as other DBs - Data size/user (50 KB) - Expected user (DAU 70万) - Update frequency - Each servers max connections - Number of servers, specs, costs - Consider when to scale servers according to user growth
- 29. Constructing Infrastructure • PerformanceVerification - Bandwith (EC2 is about 350Mbps) - Verify MongoDB (MongoDB 2.0.1、 2Shard with Replica Set、enable journaling) • READ/WRITE performance • Performance during migration • Performance through Java Driver - Performance through application
- 30. Constructing Infrastructure • Preparefor troubles - When mongod dies ... • SECONDARY becomes PRIMARY ? • Data synchronize when it restart ? • Safe when SECONDARY dies ? - When mongoc dies ... • Nothing happens ? - Succeed to restore from backup ? → no problem
- 31. Constructing Infrastructure • ReplicaSetand Shard Construction - Use large memory size - Use high speed I/O disk - Place mongoc independent from mongod - Use EBS to raise reliability (run only as SECONDARY) • Enable Journaling • Set oplogsize as 10GB
- 32. Constructing Infrastructure • Createindex in advance - Index blocks all operation ... - Took about 2 min to create index for 200,000 data (each about 50KB) - Create index during maintanance time or in background for running services
- 33. Constructing Infrastructure • ConnectionPool Tuning ※ connnection pool is for mongos property description value connectionsPerHost number of connections 100 threadsAllowedToBlockFor wait threads per 4 ConnectionMultiplier connection - set the value with nginx worker and Tomcat thread size in mind - Careful with Out of Semaphores errors. Above case would be 100 + (100 * 4) = 500
- 34. In Operation • Checkchunk balance routinely - Move chunk manually if needed • Careful when adding new Collections - Primary Shard might be overloaded because new collection data will be placed there
- 35.
- 36. mongod & mongocdied.. • Caused by virtual server host for EC2 instance. Nothing to do with MongoDB • There was NO impact for service !! • Recovered just starting mongod and mongoc processes
- 37.
- 38. Online Backup • Difficultto backup for running services • Backup while in maintenance mode • Thinking of doing backup from SECONDARY as follows, but need some verification ... 1. set balancer off 2. write lock SECONDARY and backup 3. set balance on
- 39. Upgrade • Upgrade isalso difficult for the timing • Need to do following steps 1. Stop Arbiter, upgrade, run 2. Stop SECONDARY, upgrade, run 3. Stop PRIMARY, upgrade, run
- 40. Adding Shards • Its possible to add shards later on, but have to consider it s performance for balancing - Pico is adding shards while in maintenance mode for safety
- 41. Best Chunksize • Migrationfrequently occurs, because user data was too large and chunksize was small (default 64MB) • Need to adjust chunksize according to it s data size Maybe nice if you can set the chunksize for each collections ...
- 42. Analyzing Data • Hardto analyze user data, because we handle the data in 1 document • Creating index if necessary - Tested MapReduce, but some performance problems occured. Not using it right now
- 43. Thank you ! Pleaseenjoy Animal Land !! http://apps.facebook.com/animal-land/