Your SlideShare is downloading. ×
0
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
A DevOps Perspective: MongoDB & MMF
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

A DevOps Perspective: MongoDB & MMF

582

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
582
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. MongoDB at MMFFrom a DevOps Perspective Jan 24, 2013
  • 2. Introduction! MapMyFitness!was!founded!in!2007! Offices!in!Denver,!CO!&!AusRn,!TX (w/!associates!in!SF,!Boston,!New!York,!LA,!and!Chicago)! Over!13!million!registered!users! ~80!million!geoadata!routes! (runs,!rides,!walks,!hikes,!etc)! Core!sites,!mobile!apps,!API,!whitealabel (MapMyRun,!MapMyRide,!MapMyFitness)
  • 3. MMF Platform Overview•!Python!(django)!&!PHP!(legacy!API)•!Although!MySQL!is!the!core!backing!db!for!Django,!the!majority!of!!MMF!data!lives!in!various!MongoDB!datastores.!!•!Routes!datastore!has!~120!million!objects,!currently!7TB+!of!data!!(3!member!replica!set!backed!by!a!EMC!SAN,!48GB!RAM!each)•!Django!sessions!converted!to!using!MongoDB!!!(funcRonal!scaling!example,!600M!sessions!stored)•!Live!Tracking!system!uRlizes!elasRc!replica!set!membership!to!!handle!load!scaling!for!events•!Granular!API!access/error!logging!via!json!to!MongoDB
  • 4. Route & Elevation data example (Lost on the way to MongoSeattle)
  • 5. Implementation Patterns•!Standard!Datastore!a!3!member!replica!set!!!!(small!to!med!implementaRons)•!Big!Data!implementaRon!–!sharded!cluster!(TB+)•!Buffering!Layer!a!high!memory!!!!!(load!all!data!and!index!files!into!RAM)•!Write!Heavy!a!uRlize!sharding!to!opRmize!for!writes•!Read!Heavy!a!3+n!replica!set!configuraRon!for!rapid!read!scaling!!!!(up!to!12!nodes)
  • 6. Implementation Patterns•!In!the!cloud,!tune!the!instance!type!to!the!mongo!implementaRon•!On!iron,!plan!carefully!and!dedicate!servers!completely!to!mongo!to! avoid!memory!map!contenRon•!For!DR,!spin!up!a!delayed,!hidden!replica!node!(preferably!in!a! different!datacenter)•!AggregaRon!framework!can!be!used!in!myriad!ways,!including! bridging!the!gap!to!SQL!data!warehousing!via!ETL.•!Automate!install!paSerns!for!rapid!development,!prototyping,!and! infrastructure!scaling.
  • 7. Operational Automation( example of automated mongodb install via puppet )
  • 8. Replica Set Expansion• MongoDB!is!“replicaRon!made!elegant”• Ridiculously!simple!to!add!addiRonal!members• Be!sure!to!run!IniRalSync!from!a!secondary! rs.add(!“host”!:!“livetrack_db09”,!“ini8alSync”!:!{!“state”!:!2!}!)• Both!rs.add()!and!rs.remove()!can!be!scripted!and!connected!to! Monitoring!systems!for!autoscaling
  • 9. Monitoring and Introspection•!MMS,!10gens!cloudabased!monitoring!service!(best!available)•!Supported!by!Zabbix,!Nagios,!Munin,!Server!Density,!etc•!mongostat,!mongotop,!REST!interface,!database!profiler•!Monitoring!system!triggers!can!iniRate!node!addiRons,!!removals,!service!restarts,!etc•!In!addiRon!to!servicealevel!monitoring,!use!more!advanced!!tests!to!check!for!and!alert!on!query!latency!spikes
  • 10. 10gens MMS(the one-stop shop for mongdb metrics)
  • 11. Mongo in Zabbix( Mikoomi Plugins: http://code.google.com/p/mikoomi )
  • 12. mongostat( Very useful for real-time troubleshooting )
  • 13. Operational Automation( example of automated mongodb restart action )
  • 14. Security Considerations•!MongoDB!provides!authenRcaRon!support!and!basic!permissions•!Auth!is!turned!off!by!default!to!allow!for!opRmal!performance!•!Always!run!databases!in!a!trusted!network!environment•!Lock!down!host!based!firewalls!to!limit!access!to!required!clients!•!Automate!iptables!with!puppet!or!chef,!in!EC2!use!security!groups
  • 15. Network Security Automation## Puppet Pattern for Mongodb network securityclass iptables::public { iptables::add_rule { 001 MongoDB established: rule => -A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT } iptables::add_rule { 002 MongoDB: rule => -A RH-Firewall-1-INPUT -i eth1 -p tcp -m tcp --dport 27017 -j ACCEPT } iptables::add_rule { 003 MongoDB MMF Phase II Network: rule => -A RH-Firewall-1-INPUT -i eth0 -s 172.16.16.0/20 -p tcp -m tcp --dport 27017 -j ACCEPT } iptables::add_rule { 004 MongoDB MMF Cloud Network: rule => -A RH-Firewall-1-INPUT -i eth0 -s 10.178.52.0/24 -p tcp -m tcp --dport 27017 -j ACCEPT } }
  • 16. Security Considerations•!Use!the!rule!of!leastaprivilege!to!allow!access!to!environments!•!Data!sensiRvity!should!determine!the!extent!of!security!measures•!For!nonasensiRve!data,!good!network!security!can!be!sufficient!•!In!open!environments,!be!sure!experience!matches!access!level•!Lack!of!granular!perms!allows!for!full!admin!access,!use!discreRon
  • 17. Maintenance•!Far!less!maintenance!required!than!tradiRonal!RDMBS!systems•!Regularly!perform!query!profile!analysis!and!index!audiRng•!Rebuild!databases!to!reclaim!space!lost!due!to!fragmentaRon•!Automate!checks!of!log!files!for!known!redaflags•!Regularly!review!data!throughput!rate,!storage!growth!rate,!and!!overall!business!growth!graphs!to!inform!capacity!planning.•!For!HA!tesRng,!periodically!stepadown!the!primary!to!force!failover
  • 18. Indexing Patterns or “Know Your App”• Proper!indexing!criRcal!to!performance!at!scale (monitor!slow!queries!to!catch!nonaperformant!requests)• MongoDB!is!ulRmately!flexible,!being!schemaless (mongo!gives!you!enough!rope!to!hang!yourself,!choose!wisely)• Avoid!unaindexed!queries!at!all!costs! (its!quickest!way!to!crater!your!app...!consider!aanotablescan)• Onus!on!DevOps!to!match!applicaRon!to!indexes (know!your!query!profile,!never!assume)• Shoot!for!covered!queries!wherever!possible (answer!can!be!obtained!from!indexes!only)
  • 19. Capped Collections• Use!standard!capped!collecRons!for!retaining!a!fixed!amount! of!data.!!Uses!a!FIFO!strategy!for!pruning. (based!on!data!size,!not!number!of!rows)• TTL!CollecRons!(2.2)!age!out!data!based!on!a!retenRon!Rme! configuraRon.!! (great!for!data!retenRon!requirements!of!all!types) Gotcha! Explicitly!create!the!capped!collecRon!before!any!data!is!put! into!the!system!to!avoid!autoacreaRon!of!collecRon
  • 20. Lessons Learned•!Mongo!2.2!upgrade!containing!a!capped!collecRon!created!in!1.8.4.!!This!severely!impacted! replicaRon!(RC:!no!"_id"!index,!!FIX:!add!"_id"!index)!•!Never!start!mongo!when!a!mount!point!is!missing!or!incorrectly!configured.!Mongo!may! decide!to!take!maSers!into!its!own!hands!and!resync!itself!with!the!replica!set.!!Make! sure!your!devops!and!your!hos2ng!provider!admins!are!aware!of!this•!Some!drivers!that!use!connecRon!pooling!can!freak!the!freaky!freak!when!the!primary! member!changes!(older!pymongo).!!Kicking!the!applicaRon!can!fix,!also:!upgrade!drivers•!High!locked!%!is!a!big!redaflag,!and!can!be!caused!by!a!large!number!of!simultaneous!dml! acRons!(high!insert!rate,!high!update!rate).!Consider!this!in!the!design!phase.•!Be!wary!of!automaRon!that!can!change!the!state!of!a!node!during!maintenance!mode.!! Disable!automaRon!agents!for!reduced!risk!during!criRcal!administraRve!operaRons! (filesystem!maint,!etc)
  • 21. Thank!you!chris@mapmyfitness.com

×