From MySQL to MongoDB at Wordnik (Tony Tam)

MongoSF 4/30/2010From MySQL to MongoDB,[object Object],Migrating a Live Application,[object Object],Tony Tam,[object Object]

What is Wordnik,[object Object],Project to track language ,[object Object],like GPS for English,[object Object],Dictionary is a road block to the language,[object Object],Roughly 200 new words created daily,[object Object],Language is not static,[object Object],Capture information about all words,[object Object],Meaning is often undefined in traditional sense,[object Object],Machines can determine meaning through analysis,[object Object],Needs LOTS of data,[object Object]

Why should You care,[object Object],Every Developer can use a Robust Language API!,[object Object],Wordnik migrated to MongoDB,[object Object],> 5 Billion documents,[object Object],> 1.2 TB,[object Object],Zero application downtime,[object Object],Learn from our Experience,[object Object]

Wordnik,[object Object],Not just a website!,[object Object],But we have one,[object Object],Launched Wordnik entirely on MySQL,[object Object],Hit road bumps with insert speed ~4B rows on MyISAMtables,[object Object],Tables locked for 10’s of seconds during inserts,[object Object],But we need more data!,[object Object],Created elaborate update schemes to work around it,[object Object],Lost lots of sleep babysitting servers while researching LT solution,[object Object]

Wordnik + MongoDB,[object Object],What are our storage needs?,[object Object],Database vs. Application Logic,[object Object],No PK/FK constraints,[object Object],No Stored Procedures,[object Object],Consistency?,[object Object],Lots of R&D,[object Object],Tried most all noSQL solutions,[object Object]

Migrating Storage Engines,[object Object],Many parts to this effort,[object Object],Setup & Administration,[object Object],Software Design,[object Object],Optimization,[object Object],Many types of data at Wordnik,[object Object],Corpus,[object Object],Structured HierarchicalData,[object Object],User Data,[object Object],Migrated #1 & #2,[object Object]

Server Infrastructure,[object Object],Wordnik is Heavily Read-only,[object Object],Master / Slave deployment,[object Object],Looking at replica pairs,[object Object],MongoDB loves system resources,[object Object],Wordnik runs dedicated boxes to avoid other apps being sent to disk (aka time-out),[object Object],Memory + Disk = Happy Mongo,[object Object],Many X the disk space of MySQL,[object Object],Easy pill to swallow until…,[object Object]

Server Infrastructure,[object Object],Physical Hardware,[object Object],2 x 4 core CPU, 32gb RAM, FC SAN,[object Object],Had bad luck on VMs,[object Object],(you might not),[object Object],Disk speed => performance,[object Object]

Software Design,[object Object],Two distinct use cases for MongoDB,[object Object],Identical structure, different storage engine,[object Object],Same underlying objects, same storage fidelity (largelykey/value),[object Object],Hierarchical data structure,[object Object],Same underlying objects, document-oriented storage,[object Object]

Software Design,[object Object],Create BasicDBObjects from POJOs and used collection methods,[object Object],BasicDBObjectdbo =,[object Object], new BasicDBObject("sentence",s.getSentence()),[object Object], .append("rating",s.getRating()).append(...);,[object Object],ID Generation to manage unique _ID values,[object Object],Analogous to MySQL AutoIncrement behavior,[object Object],Compatible with MySQL Ids (more later),[object Object],dbo.append("_ID", getId());,[object Object],collection.save(dbo);,[object Object],Implemented all CRUD methods in DAO,[object Object],Swappable between MongoDB and MySQL at runtime ,[object Object]

Software Design,[object Object],Key-Value storage use case,[object Object],Easy as implementing new DAOs,[object Object],SentenceHandlerh = new MongoDBSentenceHandler();,[object Object],Save methods construct BasicDBObject and call save() on collection,[object Object],Implement same interface,[object Object],Same methods against DAO between MySQL and MongoDB versions,[object Object],Data Abstraction 101,[object Object]

Software Design,[object Object],What about bulk inserts?,[object Object],FAF Queued approach,[object Object],Add objects to queue, return to caller,[object Object],Every X seconds, process queue,[object Object],All objects from same collection are appended to a single List<DBObject>,[object Object],Call collection.insert(…) before 2M characters,[object Object],Reduces network overhead,[object Object],Very fast inserts,[object Object]

Software Design,[object Object],Hierarchical Data done more elegantly,[object Object],Wordnik Dictionary Model,[object Object],Java POJOs already had JAXB annotations,[object Object],Part of public REST api,[object Object],Used Mysql,[object Object],12+ tables,[object Object],13 DAOs,[object Object],2500 lines of code,[object Object],50 requests/second uncached,[object Object],Memcache needed to maintain reasonable speed,[object Object]

Software Design,[object Object],TMGO,[object Object]

Software Design,[object Object],MongoDB’s Document Storage let us…,[object Object],Turn the Objects into JSON via Jackson Mapper (fasterxml.com),[object Object],Call save,[object Object],Support all fetch types, enhanced filters,[object Object],1000 requests / second,[object Object],No explicit caching,[object Object],No less scary code,[object Object]

Software Design,[object Object],Saving a complex object,[object Object],String rawJSON = getMapper().writeValueAsString(veryComplexObject);,[object Object],collection.save(newBasicDBOBject(getId(),JSON.parse(rawJSON));,[object Object],Fetching complex object,[object Object],BasicDBObjectdbo = cursor.next();,[object Object],ComplexObjectobj = getMapper().readValue(dbo.toString(), ComplexObject.class);,[object Object],No joins, 20x faster,[object Object]

Migrating Data,[object Object],Migrating => existing data logic,[object Object],Use logic to select DAOs appropriately,[object Object],Read from old, write with new,[object Object],Great system test for MongoDB,[object Object],SentenceHandlermysqlSh = new MySQLSentenceHandler();,[object Object],SentenceHandlermongoSh = new MongoDbSentenceHandler();,[object Object],while(hasMoreData){,[object Object],mongoSh.asyncWrite(mysqlSh.next());,[object Object], ...,[object Object],},[object Object]

Migrating Data,[object Object],Wordnik moved 5 billion rows from MySQL,[object Object],Sustained 100,000 inserts/second,[object Object],Migration tool was CPU bound,[object Object],ID generation logic, among other,[object Object],Wordnik reads MongoDB fast,[object Object],Read + create java objects @ 250k/second (!),[object Object]

Going live to Production,[object Object],Choose your use case carefully if migrating incrementally,[object Object],Scary no matter what,[object Object],Test your perf monitoring system first!,[object Object],Use your DAOs from migration,[object Object],Turn on MongoDB on one server, monitor, tune (rollback, repeat),[object Object],Full switch over when comfortable,[object Object]

Going live to Production,[object Object],Really?,[object Object],SentenceHandlerh = null;,[object Object],if(useMongoDb){,[object Object],h = new MongoDbSentenceHandler();,[object Object],},[object Object],else{,[object Object],h = new MySQLDbSentenceHandler();,[object Object],},[object Object],return h.find(...);,[object Object]

Optimizing Performance,[object Object],Home-grown connection pooling,[object Object],Master only,[object Object],ConnectionManager.getReadWriteConnection(),[object Object],Slave only,[object Object],ConnectionManager.getReadOnlyConnection(),[object Object],Round-robin all servers, bias on slaves,[object Object],ConnectionManager.getConnection(),[object Object]

Optimizing Performance,[object Object],Caching,[object Object],Had complex logic to handle cache invalidation,[object Object],Out-of-process caches are not free,[object Object],MongoDB loves your RAM,[object Object],Let it do your LRU cache (it will anyway),[object Object],Hardware,[object Object],Do not skimp on your disk or RAM,[object Object],Indexes,[object Object],Schema-less design,[object Object],Even if no values in any document, needs to read document schema to check,[object Object]

Optimizing Performance,[object Object],Disk space,[object Object],Schemaless => schema per document (row),[object Object],Choose your mappings wisely,[object Object],({veryLongAttributeName:true}) => more disk space than ({vlan:true}),[object Object]

Optimizing Performance,[object Object],A Typical Day at the Office for MongoDB,[object Object],API call rate: 47.7 calls/sec,[object Object]

Other Tips,[object Object],Data Types,[object Object],Use caution when changing,[object Object],DBObjectobj = cur.next();,[object Object],long id = (Long) obj.get(“IWasAnIntOnce”),[object Object],Attribute names,[object Object],Don’t change w/o migrating existing data!,[object Object],WTFDMDG????,[object Object]

What’s next?,[object Object],GridFS,[object Object],Store audio files on disk,[object Object],Requires clustered file system for shared access,[object Object],Capped Collections (rolling out this week),[object Object],UGC from MySQL => MongoDB,[object Object],Beg/Bribe 10gen for some Features,[object Object]

From MySQL to MongoDB at Wordnik (Tony Tam)

From MySQL to MongoDB at Wordnik (Tony Tam)

Recommended

More Related Content

What's hot

What's hot(20)

Similar to From MySQL to MongoDB at Wordnik (Tony Tam)

Similar to From MySQL to MongoDB at Wordnik (Tony Tam)(20)

More from MongoSF

More from MongoSF(20)

Recently uploaded

Recently uploaded(20)

From MySQL to MongoDB at Wordnik (Tony Tam)