Operationalizing MongoDB at AOL


Published on

MongoDC 2012

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Phone/Inperson
  • Our hardware all comes through the same door
  • MultiviewsCustom graphs, overlays, coloring, change graph typePinpoint bad releases, verify indexes
  • Operationalizing MongoDB at AOL

    1. 1. “Operationalizing” MongoDB@AOL Michael DelNegro Principal Database Administrator AOL
    2. 2. About Me• DBA at AOL (Dulles) for 6+ years• Original DBMS Background is in Sybase• Now MySQL, MongoDB, PostgreSQL and NoSQL (Neo4J, Redis, etc)• Check out www.meetup.com/NOVA- MySQL
    3. 3. The First Mongo at AOL• A MapQuest Developer in the Summer of 2010 Replaced a MySQL Datastore with Mongo• That Developer left in early 2011 :-/• Was Handed Over to Operations and Yours Truly
    4. 4. Never Forget Your First• Not Small: 40+ Hosts• Upgrade Challenges• Connection Floods• Support to the Rescue• Magnified the Reasons to Operationalize
    5. 5. Operation “Operationalize”
    6. 6. Operation “Operationalize”• Take Advantage of 10Gen Support• Establish Standards• Monitoring/Alerting• Backups• Information Sharing• Challenges
    7. 7. 10Gen Support• JIRA Documents Cases Enabling Information Sharing and Referencing• Serves Us Well in Both “Emergencies” and “Hey Quick Question”• Training Classes: Administration & Development
    8. 8. Standard Host Setup• CentOS 6• XFS/EXT4• Deadline I/O Scheduler• dirty_ratio = 10 (down from 40)• dirty_background_ratio = 5 (down from 10)• noatime
    9. 9. Standard Build Scripts• Use 64bit Linux Binaries Stored in AOL Repository• mongodb_install.sh – Parameters Include: • MongoDB Version • Replica Set Name • Sharded? (Y/N) • Port (27017 is standard) • Type (DB, Arbiter, Config, Mongos)
    10. 10. Standard Directory/Layouts• Based on our standard MySQL layout• Binaries in /opt/local/pkg• /db/mongodb/<port> base directory – Ex. /db/mongodb/27017 – Can Override • /db/mongodb/27017-repset1• Under base directory – data – etc (config files) – logs – scripts (ex stop/start)
    11. 11. ORB• AOL Technologies’ Configuration Management Database (CMDB)• Integrated with many authoritative data repositories• Unique namespace for many operations data points• Data model for operations management• Projects, Assets, People, Applications, Network Data, HCM• SQL Interface
    12. 12. Monitoring/Alerting• Argus• Nagios• Component Test Tool• MMS (MongoDB Monitoring Service)
    13. 13. Argus• Metric and Event – Collector – Thresholds – Management – Data Viewing
    14. 14. Argus• Capture serverStatus Metrics (once a minute)• Capture Replication Lag• Great for Trending• Great for Capacity Planning• Great for Troubleshooting• We Also Use for Host Metrics (CPU, I/O, etc)• Administrated By a Small and very Busy Group – Requested Additions/Changes Can Be Slow
    15. 15. Nagios• Great For Fault-detection/Alerting• Great For Show Me What Is Currently Broken• Great For Service Availability Metrics• Flexible• Reduces Pressure On NOC• Integrates With Netcool, Ignore Tool• We Write Our Own Plug-Ins
    16. 16. Component Test Tool• Runs Parallel HTTP GETs Across a Group of Hosts• Matches Regular Expressions On the Page Content (or HTTP Headers)• Displays Number (or Content) of Matches in a HTML Table
    17. 17. MMS
    18. 18. Backups• Scripts Delivered With Install• Choice Between LVM (default) and Standard (mongodump)• Ability to Backup Sharded Setup
    19. 19. Mongo @ AOL Currently
    20. 20. Mongo @ AOL Currently• 30+ Active MongoDB Projects• 500+ Servers in Production• Developed in: – Java – PHP – Ruby – C# – Erlang – Scala – Perl – C++
    21. 21. Information Sharing• Internal Wikis/Websites• UnUniversity (Organized Brown Bags)• Architecture Reviews• 10Gen JIRA Access
    22. 22. Challenges• Developers Love It• BUT Requires More Upfront Planning Than They May Realize• Should NOT Be Our Default Datastore• Better Developer OnBoarding• Aggressive Upgrading Is Required• Coordination of Driver Upgrades
    23. 23. The Future• “MongoDB As a Service”• Puppet or Chef• Internal Administration Console – Already has Visual Topologies• Internal MongoDB Operations Class – First One is in July
    24. 24. Resources• Recommended Presentation: Todd Dampier’s Rock-Solid Mongo Ops (mongolab)• Check out Jon Reed’s MongoDB for Online Advertising at AOL (MongoDB UK 2012)• www.meetup.com/Washington-DC- MongoDB-Users-Group/
    25. 25. Thank You!• www.slideshare.net/radiocats• @radiocats• www.linkedin.com/in/mdelnegro• Thank you, SeniorGif