Optimizing MongoDB: Lessons Learned at Localytics

Optimizing MongoDB:Lessons Learned at Localytics Andrew Rollins June 2011 MongoNYC

Me• Email: my first name @ localytics.com• twitter.com/andrew311• andrewrollins.com• Founder, Chief Software Architect at Localytics

Localytics• Real time analytics for mobile applications• Built on: – Scala – MongoDB – Amazon Web Services – Ruby on Rails – and more…

Why I‟m here: brain dump!• To share tips, tricks, and gotchas about: – Documents – Indexes – Fragmentation – Migrations – Hardware – MongoDB on AWS• Basic to more advanced, a compliment to MongoDB Perf Tuning at MongoSF 2011

MongoDB at Localytics• Use cases: – Anonymous loyalty information – De-duplication of incoming data• Requirements: – High throughput – Add capacity without long down-time• Scale today: – Over 1 billion events tracked in May – Thousands of MongoDB operations a second

Why MongoDB?• Stability• Community• Support• Drivers• Ease of use• Feature rich• Scale out

OPTIMIZE YOUR DATADocuments and indexes

Shorten namesBad:{ super_happy_fun_awesome_name: “yay!”}Good:{ s: “yay!”}

Use BinData for UUIDs/hashesBad:{ u: “21EC2020-3AEA-1069-A2DD-08002B30309D”, // 36 bytes plus field overhead} Good:{ u: BinData(0, “…”), // 16 bytes plus field overhead}

Override _idTurn this{ _id : ObjectId("47cc67093475061e3d95369d"), u: BinData(0, “…”) // <- this is uniquely indexed } into{ _id : BinData(0, “…”) // was the u field}Eliminated an extra index, but be careful aboutlocality... (more later, see Further Reading at end)

Pack „em in• Look for cases where you can squish multiple “records” into a single document.• Why? – Decreases number of index entries – Brings documents closer to the size of a page, alleviating potential fragmentation• Example: comments for a blog post.

Prefix IndexesSuppose you have an index on a large field, but that field doesn‟t havemany possible values. You can use a “prefix index” to greatly decreaseindex size.find({k: <kval>}){ k: BinData(0, “…”), // 32 byte SHA256, indexed }into find({p: <prefix>, k: <kval>}){ k: BinData(0, “…”), // 28 byte SHA256 suffix, not indexed p: <32-bit integer> // first 4 bytes of k packed in integer, indexed}Example: git commits

FRAGMENTATION AND MIGRATIONHidden evils

Fragmentation• Data on disk is memory mapped into RAM.• Mapped in pages (4KB usually).• Deletes/updates will cause memory fragmentation. Disk RAM doc1 doc1 find(doc1) Page deleted deleted … …

New writes mingle with old data Data doc1 PageWrite docX docX doc3 doc4 Page doc5find(docX) also pulls in old doc1, wasting RAM

Dealing with fragmentation• “mongod --repair” on a secondary, swap with primary.• 1.9 has in-place compaction, but this still holds a write-lock.• MongoDB will auto-pad records.• Pad records yourself by including and then removing extra bytes on first insert. – Alternative offered in SERVER-1810.

The Dark Side of Migrations• Chunks are a logical construct, not physical.• Shard keys have serious implications.• What could go wrong? – Let‟s run through an example.

Suppose the following Chunk 1 • K is the shard key k: 1 to 5 • K is random Chunk 2 k: 6 to 9 Shard 1 {k: 3, …} 1st write {k: 9, …} 2nd write {k: 1, …} and so on {k: 7, …} {k: 2, …} {k: 8, …}

Migrate Chunk 1 Chunk 1 k: 1 to 5 k: 1 to 5 Chunk 2 k: 6 to 9 Shard 1 Shard 2 {k: 3, …} {k: 3, …} {k: 9, …} Random IO {k: 1, …} {k: 1, …} {k: 2, …} {k: 7, …} {k: 2, …} {k: 8, …}

Shard 1 is now heavily fragmented Chunk 1 Chunk 1 k: 1 to 5 k: 1 to 5 Chunk 2 k: 6 to 9 Shard 1 Shard 2 {k: 3, …} {k: 3, …} {k: 9, …} {k: 1, …} {k: 1, …} Fragmented {k: 2, …} {k: 7, …} {k: 2, …} {k: 8, …}

Why is this scenario bad?• Random reads• Massive fragmentation• New writes mingle with old data

How can we avoid bad migrations?• Pre-split, pre-chunk• Better shard keys for better locality – Ideally where data in the same chunk tends to be in the same region of disk

Pre-split and move• If you know your key distribution, then pre-create your chunks and assign them.• See this: – http://blog.zawodny.com/2011/03/06/mongodb-pre- splitting-for-faster-data-loading-and-importing/

Better shard keys• Usually means including a time prefix in your shard key (e.g., {day: 100, id: X})• Beware of write hotspots• How to Choose a Shard Key – http://www.snailinaturtleneck.com/blog/2011/01/04/ho w-to-choose-a-shard-key-the-card-game/

OPTIMIZING HARDWARE/CLOUD

Working Set in RAM• EC2 m2.2xlarge, RAID0 setup with 16 EBS volumes.• Workers hammering MongoDB with this loop, growing data: – Loop { insert 500 byte record; find random record }• Thousands of ops per second when in RAM• Much less throughput when working set (in this case, all data and index) grows beyond RAM. Ops per second over time In RAM Not In RAM

Pre-fetch• Updates hold a lock while they fetch the original from disk.• Instead do a read to warm the doc in RAM under a shared read lock, then update.

Shard per core• Instead of a shard per server, try a shard per core.• Use this strategy to overcome write locks when writes per second matter.• Why? Because MongoDB has one big write lock.

Amazon EC2• High throughput / small working set – RAM matters, go with high memory instances.• Low throughput / large working set – Ephemeral storage might be OK. – Remember that EBS IO goes over Ethernet. – Pay attention to IO wait time (iostat). – Your only shot at consistent perf: use the biggest instances in a family.• Read this: – http://perfcap.blogspot.com/2011/03/understanding- and-using-amazon-ebs.html

Amazon EBS• ~200 seeks per second per EBS on a good day• EBS has *much* better random IO perf than ephemeral, but adds a dependency• Use RAID0• Check out this benchmark: – http://orion.heroku.com/past/2009/7/29/io_performanc e_on_ebs/• To understand how to monitor EBS: – https://forums.aws.amazon.com/thread.jspa?messag eID=124044

Further Reading• MongoDB Performance Tuning – http://www.scribd.com/doc/56271132/MongoDB-Performance-Tuning• Monitoring Tips – http://blog.boxedice.com/mongodb-monitoring/• Markus‟ manual – http://www.markus-gattol.name/ws/mongodb.html• Helpful/interesting blog posts – http://nosql.mypopescu.com/tagged/mongodb/• MongoDB on EC2 – http://www.slideshare.net/jrosoff/mongodb-on-ec2-and-ebs• EC2 and Ephemeral Storage – http://www.gabrielweinberg.com/blog/2011/05/raid0-ephemeral-storage-on-aws- ec2.html• MongoDB Strategies for the Disk Averse – http://engineering.foursquare.com/2011/02/09/mongodb-strategies-for-the-disk-averse/• MongoDB Perf Tuning at MongoSF 2011 – http://www.scribd.com/doc/56271132/MongoDB-Performance-Tuning

Thank you.• Check out Localytics for mobile analytics!• Reach me at: – Email: my first name @ localytics.com – twitter.com/andrew311 – andrewrollins.com

http://www.localytics.com	3369
http://www.dbthink.com	3005
http://nosql.mypopescu.com	854
http://blog.nosqlfan.com	671
http://ijohn.org	359
http://www.10gen.com	272
http://engineering.localytics.com	248
https://twitter.com	80
http://oakleafblog.blogspot.com	62
url_unknown	58
http://setecstudent.blogspot.com	28
http://twitter.com	26
http://feeds.feedburner.com	25
http://staging2.localytics.com	21
http://wpstaging.localytics.com	20
http://a0.twimg.com	18
http://irr.posterous.com	17
http://www.w3ccollege.org	15
http://www.slideshare.net	12
http://paper.li	11
http://cache.baidu.com	8
http://webcache.googleusercontent.com	7
http://xianguo.com	7
http://zootool.com	6
http://localytics.staging.wpengine.com	6
http://translate.googleusercontent.com	6
http://oakleafblog.blogspot.co.uk	4
https://twimg0-a.akamaihd.net	3
http://www.setecstudent.blogspot.com	3
http://wordpress.localytics.local	3
http://www.hanrss.com	2
http://newsrivr.com	2
http://www.mongodb.org	2
http://oakleafblog.blogspot.in	2
http://www.sogou.com	1
http://snapshot.soso.com	1
https://si0.twimg.com	1
http://apy.mongodb.org	1
http://localytics2.blogsblogsblogs.com	1
http://localytics.com	1
http://wix.dapulse.com	1
http://www.google.de	1
http://localhost	1
http://www.onlydoo.com	1
http://www.govloop.com	1
https://webcache.googleusercontent.com	1
http://trunk.ly	1
http://xnny.net	1
http://s.deeeki.com	1
http://v2.visualwebsiteoptimizer.com	1
http://us-w1.rockmelt.com	1

Optimizing MongoDB: Lessons Learned at Localytics

by andrew311 on Jun 07, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

51 Embeds 9,249

Statistics

Optimizing MongoDB: Lessons Learned at Localytics — Presentation Transcript