Operationalizing MongoDB at AOL

“Operationalizing” MongoDB@AOL Michael DelNegro Principal Database Administrator AOL

About Me• DBA at AOL (Dulles) for 6+ years• Original DBMS Background is in Sybase• Now MySQL, MongoDB, PostgreSQL and NoSQL (Neo4J, Redis, etc)• Check out www.meetup.com/NOVA- MySQL

The First Mongo at AOL• A MapQuest Developer in the Summer of 2010 Replaced a MySQL Datastore with Mongo• That Developer left in early 2011 :-/• Was Handed Over to Operations and Yours Truly

Never Forget Your First• Not Small: 40+ Hosts• Upgrade Challenges• Connection Floods• Support to the Rescue• Magnified the Reasons to Operationalize

Operation “Operationalize”

Operation “Operationalize”• Take Advantage of 10Gen Support• Establish Standards• Monitoring/Alerting• Backups• Information Sharing• Challenges

10Gen Support• JIRA Documents Cases Enabling Information Sharing and Referencing• Serves Us Well in Both “Emergencies” and “Hey Quick Question”• Training Classes: Administration & Development

Standard Host Setup• CentOS 6• XFS/EXT4• Deadline I/O Scheduler• dirty_ratio = 10 (down from 40)• dirty_background_ratio = 5 (down from 10)• noatime

Standard Build Scripts• Use 64bit Linux Binaries Stored in AOL Repository• mongodb_install.sh – Parameters Include: • MongoDB Version • Replica Set Name • Sharded? (Y/N) • Port (27017 is standard) • Type (DB, Arbiter, Config, Mongos)

Standard Directory/Layouts• Based on our standard MySQL layout• Binaries in /opt/local/pkg• /db/mongodb/<port> base directory – Ex. /db/mongodb/27017 – Can Override • /db/mongodb/27017-repset1• Under base directory – data – etc (config files) – logs – scripts (ex stop/start)

ORB• AOL Technologies’ Configuration Management Database (CMDB)• Integrated with many authoritative data repositories• Unique namespace for many operations data points• Data model for operations management• Projects, Assets, People, Applications, Network Data, HCM• SQL Interface

Monitoring/Alerting• Argus• Nagios• Component Test Tool• MMS (MongoDB Monitoring Service)

Argus• Metric and Event – Collector – Thresholds – Management – Data Viewing

Argus• Capture serverStatus Metrics (once a minute)• Capture Replication Lag• Great for Trending• Great for Capacity Planning• Great for Troubleshooting• We Also Use for Host Metrics (CPU, I/O, etc)• Administrated By a Small and very Busy Group – Requested Additions/Changes Can Be Slow

Nagios• Great For Fault-detection/Alerting• Great For Show Me What Is Currently Broken• Great For Service Availability Metrics• Flexible• Reduces Pressure On NOC• Integrates With Netcool, Ignore Tool• We Write Our Own Plug-Ins

Component Test Tool• Runs Parallel HTTP GETs Across a Group of Hosts• Matches Regular Expressions On the Page Content (or HTTP Headers)• Displays Number (or Content) of Matches in a HTML Table

Backups• Scripts Delivered With Install• Choice Between LVM (default) and Standard (mongodump)• Ability to Backup Sharded Setup

Mongo @ AOL Currently

Mongo @ AOL Currently• 30+ Active MongoDB Projects• 500+ Servers in Production• Developed in: – Java – PHP – Ruby – C# – Erlang – Scala – Perl – C++

Information Sharing• Internal Wikis/Websites• UnUniversity (Organized Brown Bags)• Architecture Reviews• 10Gen JIRA Access

Challenges• Developers Love It• BUT Requires More Upfront Planning Than They May Realize• Should NOT Be Our Default Datastore• Better Developer OnBoarding• Aggressive Upgrading Is Required• Coordination of Driver Upgrades

The Future• “MongoDB As a Service”• Puppet or Chef• Internal Administration Console – Already has Visual Topologies• Internal MongoDB Operations Class – First One is in July

Resources• Recommended Presentation: Todd Dampier’s Rock-Solid Mongo Ops (mongolab)• Check out Jon Reed’s MongoDB for Online Advertising at AOL (MongoDB UK 2012)• www.meetup.com/Washington-DC- MongoDB-Users-Group/

Thank You!• www.slideshare.net/radiocats• @radiocats• www.linkedin.com/in/mdelnegro• Thank you, SeniorGif

Operationalizing MongoDB at AOL

by radiocats on Jun 26, 2012

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 2

Statistics

Operationalizing MongoDB at AOL Presentation Transcript