Why we chose mongodb for guardian.co.uk

Why we chose mongodbfor guardian.co.uk Graham TackleyWeb Platform Team Lead, guardian.co.uk

“It is not the strongest of the species thatsurvives, nor the most intelligent. It is the one that is most adaptable to change.”

Early Period circa ’95The “Lash It Together” era

Early Period (95, the “Lash It Together” era) Perl, CGI, apache ExperimentalManual processesBespoke software RDBMS, scripts & static ﬁles

Mid Period circa ’00The “Vendor CMS” era

Mid Period: 2000s (The “Vendor CMS era”) Vignette / AOLserver TCL, Apache, Oracle Platform for online publishingInitially scales well withacceleration in delivery of features

Mid Period: 2000s (The “Vendor CMS era”) Surprise! Vendor’s CMSdoesn’t do what we want! Mish-mash in templates: HTML, JavaScript, TCL, SQL, PL-SQLNo model in app tier, onlyin RDBMS schema created in Oracle Designer

Mid Period: 2000s (The “Vendor CMS era”)

Mid Period: 2000s (The “Vendor CMS era”)After a few years, very difﬁcult to extend Database schemabecomes ﬁxed due to dependencies in templates

Mid Period: 2000s (The “Vendor CMS era”)If you can’t change the system:

Modern Period circa ’05-09The “J2EE Monolithic” era

Web server Web server Web server I bring you NEWS!!!App server App server App server Oracle CMS Data feeds

Web server Web server Web server Modern java app I bring you NEWS!!!App server App server App server Spring / Hibernate DDD / TDD Strong Oracle in java model Database abstracted away with ORM CMS Data feeds

Problems

Each release involves schema upgradeSchema upgrade = downtime for journalists

Complexity still increasing: 300+ tables, 10,000 lines of hibernate XML conﬁg1,000 domain objects mapped to database 70,000 lines of domain object code Very tight binding to database

ORM not really masking complexity: Database has strong inﬂuence on domain model: manydomain objects made more complex mapping joins in RDBMSComplex hibernate features used, interceptors, proxies Complex caching strategy Lots of optimisations And:We still hand code complex queries in SQL!

Load becoming an issueRDBMS difﬁcult to scale

Partial NoSQL circa ’09-10The “Sticking Plaster” era

Introduce yet more caching to patch up load problems Text Introduction of memcached

Decouple applications from database by building APIsPower APIs using alternative, more scalable technologies APIs used to scale out database reads Writes still go to RDBMs

Content APIMutualised news! http://content.guardianapis.com Read API delivered using Apache Solr Hosted in EC2 Document oriented search engine Scales well for read operations

Core Api Web servers Solr/API App server Solr/APIMemcached (20Gb) Solr/API rdbms Solr Solr/API Solr/API CMS Cloud, EC2

Mutualised news!We’ve solved our load problem (for now) but Increased our complexity

Mutualised news! We now have 3 models! RDBMS tables Java Objects JSON API

Mutualised news!

MutualisedAPI is very simple JSON news!Multiple domain concepts expressed in single document Can be designed in forwardly extensible wayWhat if the JSON API was our primary model?

Full NoSQL in developmentThe “It’s the future!” era

Database selection Simple keystore. Too simple? Huge scalability. Do we need it? Schema design difﬁcult. Simple to use, can execute similar queries to RDBMs

MongoDB Mutualised news! database Document oriented Stores parsed JSON documents Can express complex queries Can be ﬂexible about consistencyMalleable schema: can easily change at runtime Can work at both large & small scales

Flexible SchemaMutualised news!

Flexible SchemaMutualised news!Can easily represent different classes of tag as documents Both documents can be inserted into same collection Far simpler than equivalent hibernate mapped subclass conﬁguration

Flexible Schema Simple to query:Mutualised news!

Flexible Schema Simple to query:Mutualised news! Query operators: $ne, $nin, $all, $exists, $gt, $lt, $gte ...

Modifying the schemaMutualised news!

The ﬁrst project: IdentityCurrent login/registration system still in TCL/PL-SQL 3M+ users in relational database Very complex schema + PL-SQL New system required Can we migrate from Oracle to NoSql?

Build API that can support both backends Registration app guardian.co.uk API This bit is hard! Oracle

Build API that can support both backends Registration app guardian.co.uk API MongoDB Oracle

Migrate using API & decommision Registration app guardian.co.uk API MongoDB

Add new stuff! Registration app guardian.co.uk APIMongoDB Solr? Redis?

MongoDBSimple, ﬂexible schema with similar query & indexing to RDBMS Great at small or large scale Easy for developers to get going Commercial support available (10Gen) One day may power all of guardian.co.ukNo transactions / joins: developers must cater for thisProduces a net reduction in lines of code / complexity

Shameless plugs http://content.guardianapis.com We’re hiring: http://www.gnmcareers.co.uk ref JS323graham.tackley@guardian.co.uk - @tackers

http://nosql.mypopescu.com	746
http://www.myweb20.it	113
http://www.campeterson.com	81
http://hadoopbigdata.wordpress.com	48
http://cloudcomputing.by	30
http://scraps.hecticjeff.net	22
http://paper.li	19
http://www.slideshare.net	13
http://twitter.com	11
http://www.helizet.com	9
http://wiki.skygate.pl	8
http://a0.twimg.com	7
http://cf.saftas.com.tr	5
http://us-w1.rockmelt.com	5
https://nokta.theflowr.com	4
https://twitter.com	4
http://note.ocqur.com	3
http://feeds.feedburner.com	3
http://www.techgig.com	3
http://www.onlydoo.com	3
http://www.twylah.com	3
http://www.linkedin.com	2
http://www.mefeedia.com	2
http://www.linkedin-ei2.com	2
http://gbenvenuti.tumblr.com	2
http://www.hanrss.com	2
http://habrahabr.ru	1
http://slideclip.b-prep.com	1
http://esther.active8.nl	1
http://posterous.com	1
url_unknown	1
http://webcache.googleusercontent.com	1
http://feedproxy.google.com	1

Why we chose mongodb for guardian.co.uk

by Graham Tackley on Mar 21, 2011

Accessibility

Categories

Upload Details

Usage Rights

33 Embeds 1,157

Statistics

Why we chose mongodb for guardian.co.uk Presentation Transcript