Moving from Relational to Document Store

8,572 views

Published on

guardian.co.uk is a leading UK-based news website. We've spent ten years fighting relational database representations of our domain model, until the implementation of our API made us realise that if only we could store documents everything got simpler. I'll talk about the history that led us to choosing MongoDB as a key part of our infrastructure going forward and how we're progressively migrating from our relational database.

With huge credit to Mat Wall @matwall for creating the original version of this talk.

Published in: Technology
  • Be the first to comment

Moving from Relational to Document Store

  1. 1. Evolving from Relational toDocument Store Graham TackleyWeb Platform Team Lead, guardian.co.uk
  2. 2. “It is not the strongest of the species thatsurvives, nor the most intelligent. It is the one that is most adaptable to change.”
  3. 3. Early Period circa ’95The “Lash It Together” era
  4. 4. Early Period (95, the “Lash It Together” era) Perl, CGI, apache ExperimentalManual processesBespoke software RDBMS, scripts & static files
  5. 5. Mid Period circa ’00The “Vendor CMS” era
  6. 6. Mid Period: 2000s (The “Vendor CMS era”) Vignette / AOLserver TCL, Apache, Oracle Platform for online publishingInitially scales well withacceleration in delivery of features
  7. 7. Mid Period: 2000s (The “Vendor CMS era”) Surprise! Vendor’s CMSdoesn’t do what we want! Mish-mash in templates: HTML, JavaScript, TCL, SQL, PL-SQLNo model in app tier, onlyin RDBMS schema created in Oracle Designer
  8. 8. Mid Period: 2000s (The “Vendor CMS era”)
  9. 9. Mid Period: 2000s (The “Vendor CMS era”)
  10. 10. Mid Period: 2000s (The “Vendor CMS era”)After a few years, very difficult to extend Database schemabecomes fixed due to dependencies in templates
  11. 11. Mid Period: 2000s (The “Vendor CMS era”)If you can’t change the system:
  12. 12. Modern Period circa ’05-09The “J2EE Monolithic” era
  13. 13. Web server Web server Web server I bring you NEWS!!!App server App server App server Oracle CMS Data feeds
  14. 14. Web server Web server Web server Modern java app I bring you NEWS!!!App server App server App server Spring / Hibernate DDD / TDD Strong Oracle in java model Database abstracted away with ORM CMS Data feeds
  15. 15. Problems
  16. 16. Each release involves schema upgradeSchema upgrade = downtime for journalists
  17. 17. Complexity still increasing: 300+ tables, 10,000 lines of hibernate XML config1,000 domain objects mapped to database 70,000 lines of domain object code Very tight binding to database
  18. 18. ORM not really masking complexity: Database has strong influence on domain model: manydomain objects made more complex mapping joins in RDBMSComplex hibernate features used, interceptors, proxies Complex caching strategy
  19. 19. ORM not really masking complexity: Database has strong influence on domain model: manydomain objects made more complex mapping joins in RDBMSComplex hibernate features used, interceptors, proxies Complex caching strategy And: We still hand code most queries in SQL!
  20. 20. Load becoming an issueRDBMS difficult to scale
  21. 21. Partial NoSQL circa ’09-10The “Sticking Plaster” era
  22. 22. Introduce yet more caching to patch up load problems Text Introduction of memcached
  23. 23. Decouple applications from database by building APIsPower APIs using alternative, more scalable technologies APIs used to scale out database reads Writes still go to RDBMs
  24. 24. Mutualised news! Content API http://content.guardianapis.comRead only access to 10 years worth of content
  25. 25. Core Api Web servers Solr/API App servers Solr/APIMemcached (20Gb) Solr/API oracle Solr Solr/API Solr/API CMS Cloud, EC2
  26. 26. Mutualised news!We’ve solved our load problem (for now) but Increased our complexity
  27. 27. Mutualised news! We now have 3 models! RDBMS tables Java Objects JSON API
  28. 28. Mutualised news!
  29. 29. Mutualised news!
  30. 30. Mutualised news!
  31. 31. Mutualised news! understandable JSON API is simple andMultiple domain concepts expressed in single document Can be designed in forwardly extensible way
  32. 32. Mutualised news! understandable JSON API is simple andMultiple domain concepts expressed in single document Can be designed in forwardly extensible wayWhat if the JSON API was our primary model?
  33. 33. Full NoSQL in developmentThe “It’s the future!” era
  34. 34. MongoDB Mutualised news! database Document oriented Stores parsed JSON documents Can express complex queries Can be flexible about consistencyMalleable schema: can easily change at runtime Can work at both large & small scales
  35. 35. Flexible SchemaMutualised news!
  36. 36. Flexible SchemaMutualised news!
  37. 37. Flexible SchemaMutualised news!Can easily represent different classes of tag as documents Both documents can be inserted into same collection Far simpler than equivalent hibernate mapped subclass configuration
  38. 38. Flexible Schema Simple to query:Mutualised news!
  39. 39. Flexible Schema Simple to query:Mutualised news! Query operators: $ne, $nin, $all, $exists, $gt, $lt, $gte ...
  40. 40. Modifying the schemaMutualised news!
  41. 41. Modifying the schemaMutualised news!
  42. 42. Modifying the schemaMutualised news!
  43. 43. The first project: IdentityCurrent login/registration system still in TCL/PL-SQL 3M+ users in relational database Very complex schema + PL-SQL New system required Can we migrate from Oracle to NoSql?
  44. 44. Build API that can support both backends Registration app guardian.co.uk API This bit is hard! Oracle
  45. 45. Build API that can support both backends Registration app guardian.co.uk API MongoDB Oracle
  46. 46. Migrate using API & decommision Registration app guardian.co.uk API MongoDB
  47. 47. Add new stuff! Registration app guardian.co.uk APIMongoDB Solr? Redis?
  48. 48. Summary: Evolving to Document Store Relational databases are dead Ruthlessly reject relational complexity Think hard about your json model Plan your evolution http://content.guardianapis.com graham.tackley@guardian.co.uk - @tackers

×