Mistakes Made and Lessons Learnt Scaling Plone post-Launch

  • 1,568 views
Uploaded on

This is a run down of the mistakes we made and the lessons we learnt trying to scale a big Plone site with un-anticipated traffic once the site had gone live and the budget all been spent. Find out …

This is a run down of the mistakes we made and the lessons we learnt trying to scale a big Plone site with un-anticipated traffic once the site had gone live and the budget all been spent. Find out what we did wrong, and how we fixed it.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,568
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
6
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Matt Hamilton & Matt Sital-SinghPlone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 2. 0Plone Conference 2011San Francisco Background Matt Hamilton Matt Sital-Singh
  • 3. Use Case • Learning Management-type system • A lot of content • Users ‘editing’ content, but not necessarily aware of it • Lots of ‘clustering’ of load on resourcesPlone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 4. Scale • Plone 3 • User accounts: 30,000 • Messages: 160,000 • Course questions: 260,000 • Other: 60,000Plone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 5. 1 Don’t Create Ghost ContentPlone Conference 2011San Francisco Matt Hamilton Matt Sital-Singh
  • 6. • User perception of speed • If they expect a ‘read’ they want a ‘read’ - user signups - course starting • Will accept a delay if you are actually processing somethingPlone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 7. 2 Use Optimization ProductsPlone Conference 2011San Francisco Matt Hamilton Matt Sital-Singh
  • 8. • experimental.catalogqueryplan • experimental.contentcreation • experimental.daterangeindexoptimisations • experimental.aggressiveopaquespeedup • collective.indexingPlone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 9. 3 One Plone site or Multiple?Plone Conference 2011San Francisco Matt Hamilton Matt Sital-Singh
  • 10. • One Plone site per ‘site’ • Multiple ‘sites’ per Plone site • Common content?Plone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 11. 4 Large Catalogs are SlowPlone Conference 2011San Francisco Matt Hamilton Matt Sital-Singh
  • 12. • We had 260,000 questions • each had 2-4 answers • = 520,000 - 1,400,000 answers All in portal_catalog!Plone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 13. 5 Optimize your CatalogsPlone Conference 2011San Francisco Matt Hamilton Matt Sital-Singh
  • 14. • Pre-load Catalog Queryplan • Balance your BTreesPlone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 15. effective date index: 1308585257 1319126057 1319730857 1319817170 1319817198Plone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 16. 1308585257 1319126057 1319730857 1319817170 1319817198Plone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 17. • Used catalog multiplexer to assign ‘Answer’ content type to dummy catalog • Moved messages to separate message_catalog • Monkey-patch items out of the catalog?Plone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 18. 6 If You Know Where Something Is - Don’t Search for itPlone Conference 2011San Francisco Matt Hamilton Matt Sital-Singh
  • 19. • Got into bad habit of using the catalog ALL the time • Don’t just blindly use catalog when unrestrictedTraverse / contentValues will do • RAM is cheap, user patience is NOTPlone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 20. 7 Use unrestrictedSearchResults if possiblePlone Conference 2011San Francisco Matt Hamilton Matt Sital-Singh
  • 21. • unrestrictedSearchResults does not check: • effective index • allowed_roles_and_users index ➡ A lot less work for the catalog to doPlone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 22. 8Is Plone the Right Tool?Plone Conference 2011San Francisco Matt Hamilton Matt Sital-Singh
  • 23. • “When you have a hammer, everything looks like a nail...” • A lot of this data is relational... • Not much of it is CMSish • Maybe should have used Pyramid?Plone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 24. 9 Use Event Driven Workflow (Don’t recalculate)Plone Conference 2011San Francisco Matt Hamilton Matt Sital-Singh
  • 25. • User rating / Course ranking • Use workflow event subscribers, don’t re- calculate on view every timePlone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 26. 10 Some things don’t need to be immediatePlone Conference 2011San Francisco Matt Hamilton Matt Sital-Singh
  • 27. • “You have 10 new messages” • Not everything has to be updated all the time in every location on the sitePlone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh
  • 28. Thanks!Plone Conference 2011 Matt HamiltonSan Francisco Matt Sital-Singh