2013MongoDB Hacks ofFrustrationFoursquare Hacks for a Better MongoMongoNYCJune 21, 2013Leo KimSoftware Engineer, Foursquare
2013Agenda• About Foursquare• Vital stats• Our hacks/tools• Questions2013
2013What is Foursquare?Foursquare helps youexplore the worldaround you.Meet up with friends,discover new places,and save m...
2013Big stats35,000,000+ people4,000,000,000+ check-ins55,000,000+ points of interest1,300,000+ merchants2013
2013Moar stats●5MM-6MM checkins a day●~4K-5K qps against our API, ~150K-300K qps againstMongo●11 clusters, 8 sharded + 3 r...
2013Mongo has scaled with usWe have been using Mongo for three years.Mongo has enabled us to scale our servicealong with o...
2013Still, some things to deal with●Monitoring– MMS is good, but always could use more stats tonarrow down on pain points....
2013Monitoring hack: “ODash”2013
2013Monitoring hack: “Mongolyzer”2013
2013Monitoring hack: “Telemetry”2013
2013Data size and fragmentation• Problem: Even with bare metal and SSDs, fragmentationcan degrade performance by increasin...
2013Alerts!2013
2013Hack: “Mackinac”• (Mostly) automated repair script• “Kill file” mongod– Drain queries gracefully from mongod– About ki...
2013Hack: “Mackinac”2013
2013Hack: Shard key checker• Checks for shard key usage in the app• Loads shard keys from mongo config servers,matches key...
2013Detect all-shards queries!• Problem: All-shards queries– Not all our queries use shard keys (unfortunately)– Use up co...
2013All-shards queries[2013-06-09 19:24:27,706] WARN c.f.boot.RogueShardKeyChecker - Possible all-shards query: db.venues....
2013All-shards queries2013
2013Find hot chunks!• Problem: Autobalancer balances by data size, but notload.– Checkins shard key → {u : 1}– Imagine a g...
2013Hack: Hot chunk detector• Need to do a little more to make this work with theshard key checker– Create trees of chunk ...
2013Hack: Hot chunk detector2013
2013Deeper hack: Hot chunk mover• A standalone process reading from JSON endpoint onhot chunk detector– Identifies the hot...
2013Sample hot chunk detector json2013
2013Fix data integrity!• Problem: Application doesnt always clean up after itselfproperly.– Duplicate documents can exist ...
2013Hack: “Chunksanity”• Simple algorithm:– Connects to each shard– Iterates through each document in eachcollection– Veri...
2013Sample Chunksanity logging[2013-05-29 18:23:36,128] [main] INFO c.f.m.chunksanity.MongoChunkSanity -Logging misplaced ...
20132013Questions?leo@foursquare.comfoursquare.com/jobs2013
MongoDB Hacks of Frustration
MongoDB Hacks of Frustration
Upcoming SlideShare
Loading in...5
×

MongoDB Hacks of Frustration

2,245

Published on

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,245
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
25
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

MongoDB Hacks of Frustration

  1. 1. 2013MongoDB Hacks ofFrustrationFoursquare Hacks for a Better MongoMongoNYCJune 21, 2013Leo KimSoftware Engineer, Foursquare
  2. 2. 2013Agenda• About Foursquare• Vital stats• Our hacks/tools• Questions2013
  3. 3. 2013What is Foursquare?Foursquare helps youexplore the worldaround you.Meet up with friends,discover new places,and save moneyusing your phone.
  4. 4. 2013Big stats35,000,000+ people4,000,000,000+ check-ins55,000,000+ points of interest1,300,000+ merchants2013
  5. 5. 2013Moar stats●5MM-6MM checkins a day●~4K-5K qps against our API, ~150K-300K qps againstMongo●11 clusters, 8 sharded + 3 replica sets– Mongo 2.2.{3,4}●~4TB of data and indexes, all of it kept in memory– 24-core Intel, 192GB RAM, 4 120GB SSD drives●Extensive use of sharding + replica setReadPreference.secondary reads2013
  6. 6. 2013Mongo has scaled with usWe have been using Mongo for three years.Mongo has enabled us to scale our servicealong with our growing user base. It hasgiven us the flexibility and agility to innovatein an exciting space where there is still muchto figure out.2013
  7. 7. 2013Still, some things to deal with●Monitoring– MMS is good, but always could use more stats tonarrow down on pain points.●General maintenance– Constant struggle with data fragmentation●Sharding– No load-based balancing (SERVER-2472)– Overhead of all-shards queries– Bugs can leave duplicate records on the wrongshards2013
  8. 8. 2013Monitoring hack: “ODash”2013
  9. 9. 2013Monitoring hack: “Mongolyzer”2013
  10. 10. 2013Monitoring hack: “Telemetry”2013
  11. 11. 2013Data size and fragmentation• Problem: Even with bare metal and SSDs, fragmentationcan degrade performance by increasing data size beyondavailable memory.– Can also be an issue with autobalancing as chunkmoves induce increased paging and further degradeI/O• We have enough replicas (~400) where we need to dothis regularly2013
  12. 12. 2013Alerts!2013
  13. 13. 2013Hack: “Mackinac”• (Mostly) automated repair script• “Kill file” mongod– Drain queries gracefully from mongod– About kill files:http://engineering.foursquare.com/2012/06/20/stability-in-the-midst-of-chaos/• Stops mongod• Resyncs from primary– We considered running compact() but it doesntreclaim disk space. May revisit this though.2013
  14. 14. 2013Hack: “Mackinac”2013
  15. 15. 2013Hack: Shard key checker• Checks for shard key usage in the app• Loads shard keys from mongo config servers,matches keys against a given {query, document}• e.g.db.users({ _id : 12345 }) // shard key match!db.users({ twitter: “joe@some.com” }) // shard keymiss!• Why use this?2013
  16. 16. 2013Detect all-shards queries!• Problem: All-shards queries– Not all our queries use shard keys (unfortunately)– Use up connections, network traffic, overhead inquery processing on mongod + mongos– Gets worse with more shards– What happens if one of the replicas is notresponding?●Solution: Measure by intercepting queries withshard key checker and count misses2013
  17. 17. 2013All-shards queries[2013-06-09 19:24:27,706] WARN c.f.boot.RogueShardKeyChecker - Possible all-shards query: db.venues.find({ "del" : { "$ne" : true} , "mayor" : xxxx , "closed" :{ "$ne" : true}}).sort({ "mayor_count" : -1})[2013-06-09 19:24:28,296] WARN c.f.boot.RogueShardKeyChecker - Possible all-shards query: db.tips.find({ "uid" : xxxx}, { "_id" : 1})[2013-06-09 19:24:28,326] WARN c.f.boot.RogueShardKeyChecker - Possible all-shards query: db.photos.find({ "uid" : xxxx})[2013-06-09 19:24:28,696] WARN c.f.boot.RogueShardKeyChecker - Possible all-shards query: db.comments2.find({ "c.u" : xxxx})[2013-06-09 19:24:32,246] WARN c.f.boot.RogueShardKeyChecker - Possible all-shards query: db.user_venue_aggregations2.find({ "_id" : { "$gte" : { "u" : xxxx , "v" :{ "$oid" : "000000000000000000000000"}} , "$lte" : { "u" : xxxx , "v" : { "$oid" :"000000000000000000000000"}}}})2013
  18. 18. 2013All-shards queries2013
  19. 19. 2013Find hot chunks!• Problem: Autobalancer balances by data size, but notload.– Checkins shard key → {u : 1}– Imagine a group of users who check in a bunch.– Imagine the balancer putting all those users on thesame shard, or even the same chunk.• Solution: Intercept queries with shard key checker andbucket hits by chunk2013
  20. 20. 2013Hack: Hot chunk detector• Need to do a little more to make this work with theshard key checker– Create trees of chunk ranges per-collection– Match shard keys from queries to chunk ranges,accumulate counts2013
  21. 21. 2013Hack: Hot chunk detector2013
  22. 22. 2013Deeper hack: Hot chunk mover• A standalone process reading from JSON endpoint onhot chunk detector– Identifies the hottest chunk– Attempts to split it (if necessary)– Move the chunk to the “coldest” shard• Subject to same problems as regular chunk moves, candisrupt latencies• Currently using a hit ratio of p9999/p50 to identify hotchunk• Work in progress2013
  23. 23. 2013Sample hot chunk detector json2013
  24. 24. 2013Fix data integrity!• Problem: Application doesnt always clean up after itselfproperly.– Duplicate documents can exist on multiple shards• Solution: Compare the document shard key from thehost replica against the canonical metadata in shardkey checker (i.e. where the document should “live”)2013
  25. 25. 2013Hack: “Chunksanity”• Simple algorithm:– Connects to each shard– Iterates through each document in eachcollection– Verifies that the document is correctly placedaccording to chunk data in mongo config server– Deletes any incorrectly placed documents• Heavyweight process, run only periodically onspecific suspicious collections2013
  26. 26. 2013Sample Chunksanity logging[2013-05-29 18:23:36,128] [main] INFO c.f.m.chunksanity.MongoChunkSanity -Logging misplaced docs to localhost/production_misplaced_docs[2013-05-29 18:23:36,301] [main] INFO c.f.m.chunksanity.MongoChunkSanity -Verifying chunks on users at office-canary-2/10.101.1.175:26190[2013-05-29 18:23:37,971] [ForkJoinPool-1-worker-1] INFOc.f.m.chunksanity.MongoChunkSanity - Looking at collection foursquare.users onshard shard0007 using filter: { } and shardKey { "_id" : 1.0}[2013-05-29 18:24:06,892] [ForkJoinPool-1-worker-2] INFOc.f.m.chunksanity.MongoChunkSanity - Done with foursquare.users on shard users200/xxxx misplaced in 28911 ms2013
  27. 27. 20132013Questions?leo@foursquare.comfoursquare.com/jobs2013
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×