• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Operationalizing MongoDB at AOL

Operationalizing MongoDB at AOL



MongoDC 2012

MongoDC 2012



Total Views
Views on SlideShare
Embed Views



2 Embeds 3

http://us-w1.rockmelt.com 2
http://www.docshut.com 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Phone/Inperson
  • Our hardware all comes through the same door
  • MultiviewsCustom graphs, overlays, coloring, change graph typePinpoint bad releases, verify indexes

Operationalizing MongoDB at AOL Operationalizing MongoDB at AOL Presentation Transcript

  • “Operationalizing” MongoDB@AOL Michael DelNegro Principal Database Administrator AOL
  • About Me• DBA at AOL (Dulles) for 6+ years• Original DBMS Background is in Sybase• Now MySQL, MongoDB, PostgreSQL and NoSQL (Neo4J, Redis, etc)• Check out www.meetup.com/NOVA- MySQL
  • The First Mongo at AOL• A MapQuest Developer in the Summer of 2010 Replaced a MySQL Datastore with Mongo• That Developer left in early 2011 :-/• Was Handed Over to Operations and Yours Truly
  • Never Forget Your First• Not Small: 40+ Hosts• Upgrade Challenges• Connection Floods• Support to the Rescue• Magnified the Reasons to Operationalize
  • Operation “Operationalize”
  • Operation “Operationalize”• Take Advantage of 10Gen Support• Establish Standards• Monitoring/Alerting• Backups• Information Sharing• Challenges
  • 10Gen Support• JIRA Documents Cases Enabling Information Sharing and Referencing• Serves Us Well in Both “Emergencies” and “Hey Quick Question”• Training Classes: Administration & Development
  • Standard Host Setup• CentOS 6• XFS/EXT4• Deadline I/O Scheduler• dirty_ratio = 10 (down from 40)• dirty_background_ratio = 5 (down from 10)• noatime
  • Standard Build Scripts• Use 64bit Linux Binaries Stored in AOL Repository• mongodb_install.sh – Parameters Include: • MongoDB Version • Replica Set Name • Sharded? (Y/N) • Port (27017 is standard) • Type (DB, Arbiter, Config, Mongos)
  • Standard Directory/Layouts• Based on our standard MySQL layout• Binaries in /opt/local/pkg• /db/mongodb/<port> base directory – Ex. /db/mongodb/27017 – Can Override • /db/mongodb/27017-repset1• Under base directory – data – etc (config files) – logs – scripts (ex stop/start)
  • ORB• AOL Technologies’ Configuration Management Database (CMDB)• Integrated with many authoritative data repositories• Unique namespace for many operations data points• Data model for operations management• Projects, Assets, People, Applications, Network Data, HCM• SQL Interface
  • Monitoring/Alerting• Argus• Nagios• Component Test Tool• MMS (MongoDB Monitoring Service)
  • Argus• Metric and Event – Collector – Thresholds – Management – Data Viewing
  • Argus• Capture serverStatus Metrics (once a minute)• Capture Replication Lag• Great for Trending• Great for Capacity Planning• Great for Troubleshooting• We Also Use for Host Metrics (CPU, I/O, etc)• Administrated By a Small and very Busy Group – Requested Additions/Changes Can Be Slow
  • Nagios• Great For Fault-detection/Alerting• Great For Show Me What Is Currently Broken• Great For Service Availability Metrics• Flexible• Reduces Pressure On NOC• Integrates With Netcool, Ignore Tool• We Write Our Own Plug-Ins
  • Component Test Tool• Runs Parallel HTTP GETs Across a Group of Hosts• Matches Regular Expressions On the Page Content (or HTTP Headers)• Displays Number (or Content) of Matches in a HTML Table
  • MMS
  • Backups• Scripts Delivered With Install• Choice Between LVM (default) and Standard (mongodump)• Ability to Backup Sharded Setup
  • Mongo @ AOL Currently
  • Mongo @ AOL Currently• 30+ Active MongoDB Projects• 500+ Servers in Production• Developed in: – Java – PHP – Ruby – C# – Erlang – Scala – Perl – C++
  • Information Sharing• Internal Wikis/Websites• UnUniversity (Organized Brown Bags)• Architecture Reviews• 10Gen JIRA Access
  • Challenges• Developers Love It• BUT Requires More Upfront Planning Than They May Realize• Should NOT Be Our Default Datastore• Better Developer OnBoarding• Aggressive Upgrading Is Required• Coordination of Driver Upgrades
  • The Future• “MongoDB As a Service”• Puppet or Chef• Internal Administration Console – Already has Visual Topologies• Internal MongoDB Operations Class – First One is in July
  • Resources• Recommended Presentation: Todd Dampier’s Rock-Solid Mongo Ops (mongolab)• Check out Jon Reed’s MongoDB for Online Advertising at AOL (MongoDB UK 2012)• www.meetup.com/Washington-DC- MongoDB-Users-Group/
  • Thank You!• www.slideshare.net/radiocats• @radiocats• www.linkedin.com/in/mdelnegro• Thank you, SeniorGif