9. Using Cloud Services
• Amazon Web Services
- EC2 (Instance Store + EBS)
- S3
- CloudFront
- Route53
- Elastic Load Balancing
• Google Apps
• Kontagent
10. System Relations
iframe API Call Callback for
Payment API
Web
HTML
Server
JSON
Flash
Command
AMF Server
AMF : Actionscript Message Format
11. L m1.large
Servers XX
EBS
m2.2xlarge
EBS
ELB ELB
S3
Web L 3 Command L 4 CloudFront
nginx nginx Route 53
Tomcat Tomcat
バッチ L
mongos mongos
バッチ
MySQL
Shard 5 MySQL L memcached L 2
monitor L
MongoDB XX MySQL memcached
EBS munin
mongod
MySQL L nagios
MongoDB XX
MySQL admin L ビルド L
mongod EBS
nginx redmine
MongoDB XX MongoDB XX 3 Tomcat jenkins
mongod mongoc
EBS EBS mongos Maven SVN
EBS
Secondary
15. • Used in Ameba Pico
http://apps.facebook.com/amebapico/
16. • I like MongoDB features !!
- ReplicaSet
- Auto-Sharding
- Can handle complex data structures
- Index, Advanced Queries
- Developed actively
- Quite easy to understand
17. • Fits in Animal Land requirements
- Complex data structures (City s grid
data, user/structure parameters, ...)
- Most processes run sequencially and
doesn t update same data at a time
- Maintenance-less
• Change data structures
dynamically
• Durability, Scalability
18. • Resolve MongoDB s weak points by
other way
- Some data are stored in other DB
• Payments history needs reliability,
so they are stored in MySQL
• Temporary data are stored in
memcached
- Don t think of using transactions
20. Developing Application
• Define data without using transaction
- User data are defined in 1 document
and updated only once in block
User Data
{ facebookId : xxx,
Image
status : { lv : 10, coin : 9999, ... },
layerInfo : 1¦1¦0¦1¦2¦1¦1¦3¦1¦1¦4¦0... ,
structure : {
1¦1 : { id : xxxx, depth : 3, width : 3, aspect : 1, ... }, ...
},
inventory : [ { id : xxx, size : xxx, ... }, ... ],
neighbor : [ { id : xxx, newFlg : false, ... }, ... ],
animal : [ { id : xxx, color : 0, pos : 20¦20 , ... } ],
...
}
21. Developing Application
• Cutdown data size as possible
- City s grid data are stored in 1 field
(Expanded when application uses)
- Data in design phase(500 KB)
layerInfo : {
1¦1 : 0,
1¦2 : 1,
....
}
- Current data(50 KB)
layerInfo : 1¦1¦0¦1¦2¦1¦1¦3¦1¦1¦4¦0...
22. Developing Application
• Careful not to define too many fields
- It took more than 5 sec for find()
when data contained more than
20,000 fields (144x144 City s grid
data)
- Consider data size, and also for the
time to parse BSON
23. Developing Application
• Shard Key is decided in following
policy
- Don t use low-cardinality key
- Recent used data should be on
memory, and data not used should
not be
- Frequently read/write data should be
in each shard in nice balance
24. Developing Application
• Use targeted operation as possible
- Use Shard Key
- Use index for non Shard Key op
Operation Type
db.foo.find({ShardKey : 1}) Targeted
db.foo.find({ShardKey : 1, NonShardKey : 1}) Targeted
db.foo.find({NonShardKey : 1}) Global
db.foo.find() Global
db.foo.insert(<object>) Targeted
db.foo.update({ShardKey : 1}, <object>) Targeted
db.foo.remove({ShardKey : 1})
db.foo.update({NonShardKey : 1}, <object>) Global
db.foo.remove({NonShardKey : 1})
25. Developing Application
• Decrease update frequency
- Store master data in memory
- Queue mechanism in Flash
- User operations are processed in
block (once per 5 sec)
• Server processes sequencially
Processes
User op. Stored in
Sequencially
Queue
Queue
Command
Flash
Server
Send in Block
once per 5 sec
26. Developing Application
• Develop efficiently using O/R Mapper
- Spring Data - MongoDB and
Wrapper classes
@Autowired
protected MongoTemplate mongoTemplate;
public void insert(T entity) {
mongoTemplate.save(entity);
}
- Can use any serializable object
- Maybe easier to deal with than RDB
O/R Mapper
27. Developing Application
• Implements without expecting rollback
- Give up for some inconsistent data
- Careful not to inflict disadvantages
to user
28. Constructing Infrastructure
• Estimate same as other DBs
- Data size/user (50 KB)
- Expected user (DAU 70万)
- Update frequency
- Each servers max connections
- Number of servers, specs, costs
- Consider when to scale servers
according to user growth
29. Constructing Infrastructure
• Performance Verification
- Bandwith (EC2 is about 350Mbps)
- Verify MongoDB (MongoDB 2.0.1、
2Shard with Replica Set、enable
journaling)
• READ/WRITE performance
• Performance during migration
• Performance through Java Driver
- Performance through application
30. Constructing Infrastructure
• Prepare for troubles
- When mongod dies ...
• SECONDARY becomes PRIMARY ?
• Data synchronize when it restart ?
• Safe when SECONDARY dies ?
- When mongoc dies ...
• Nothing happens ?
- Succeed to restore from backup ?
→ no problem
31. Constructing Infrastructure
• ReplicaSet and Shard Construction
- Use large memory size
- Use high speed I/O disk
- Place mongoc independent from
mongod
- Use EBS to raise reliability (run only
as SECONDARY)
• Enable Journaling
• Set oplogsize as 10GB
32. Constructing Infrastructure
• Create index in advance
- Index blocks all operation ...
- Took about 2 min to create index for
200,000 data (each about 50KB)
- Create index during maintanance
time or in background for running
services
33. Constructing Infrastructure
• Connection Pool Tuning
※ connnection pool is for mongos
property description value
connectionsPerHost number of connections 100
threadsAllowedToBlockFor wait threads per
4
ConnectionMultiplier connection
- set the value with nginx worker and
Tomcat thread size in mind
- Careful with Out of Semaphores
errors. Above case would be
100 + (100 * 4) = 500
34. In Operation
• Check chunk balance routinely
- Move chunk manually if needed
• Careful when adding new Collections
- Primary Shard might be overloaded
because new collection data will be
placed there
36. mongod & mongoc died..
• Caused by virtual server host for EC2
instance. Nothing to do with MongoDB
• There was NO impact for service !!
• Recovered just starting mongod and
mongoc processes
38. Online Backup
• Difficult to backup for running services
• Backup while in maintenance mode
• Thinking of doing backup from
SECONDARY as follows, but need
some verification ...
1. set balancer off
2. write lock SECONDARY and backup
3. set balance on
39. Upgrade
• Upgrade is also difficult for the timing
• Need to do following steps
1. Stop Arbiter, upgrade, run
2. Stop SECONDARY, upgrade, run
3. Stop PRIMARY, upgrade, run
40. Adding Shards
• It s possible to add shards later on,
but have to consider it s performance
for balancing
- Pico is adding shards while in
maintenance mode for safety
41. Best Chunksize
• Migration frequently occurs, because
user data was too large and chunksize
was small (default 64MB)
• Need to adjust chunksize according to
it s data size
Maybe nice if you can set the chunksize
for each collections ...
42. Analyzing Data
• Hard to analyze user data, because we
handle the data in 1 document
• Creating index if necessary
- Tested MapReduce, but some
performance problems occured. Not
using it right now
43. Thank you !
Please enjoy Animal Land !!
http://apps.facebook.com/animal-land/