SlideShare a Scribd company logo
Matt Hamilton
                          &
                   Matt Sital-Singh

Plone Conference 2011                 Matt Hamilton
San Francisco                         Matt Sital-Singh
0
Plone Conference 2011
San Francisco
                        Background


                                     Matt Hamilton
                                     Matt Sital-Singh
Use Case

        • Learning Management-type system
        • A lot of content
        • Users ‘editing’ content, but not necessarily
            aware of it
        • Lots of ‘clustering’ of load on resources
Plone Conference 2011                              Matt Hamilton
San Francisco                                      Matt Sital-Singh
Scale
        • Plone 3
        • User accounts: 30,000
        • Messages: 160,000
        • Course questions: 260,000
        • Other: 60,000
Plone Conference 2011                 Matt Hamilton
San Francisco                         Matt Sital-Singh
1     Don’t Create Ghost
                 Content

Plone Conference 2011
San Francisco
                             Matt Hamilton
                             Matt Sital-Singh
• User perception of speed
        • If they expect a ‘read’ they want a ‘read’
            - user signups
            - course starting

        • Will accept a delay if you are actually
            processing something


Plone Conference 2011                               Matt Hamilton
San Francisco                                       Matt Sital-Singh
2         Use Optimization
                    Products

Plone Conference 2011
San Francisco
                                   Matt Hamilton
                                   Matt Sital-Singh
• experimental.catalogqueryplan
        • experimental.contentcreation
        • experimental.daterangeindexoptimisations
        • experimental.aggressiveopaquespeedup
        • collective.indexing
Plone Conference 2011                          Matt Hamilton
San Francisco                                  Matt Sital-Singh
3        One Plone site or
                  Multiple?

Plone Conference 2011
San Francisco
                                   Matt Hamilton
                                   Matt Sital-Singh
• One Plone site per ‘site’
        • Multiple ‘sites’ per Plone site
        • Common content?


Plone Conference 2011                       Matt Hamilton
San Francisco                               Matt Sital-Singh
4       Large Catalogs are
                    Slow

Plone Conference 2011
San Francisco
                                   Matt Hamilton
                                   Matt Sital-Singh
• We had 260,000 questions
        • each had 2-4 answers
        • = 520,000 - 1,400,000 answers
                        All in portal_catalog!

Plone Conference 2011                            Matt Hamilton
San Francisco                                    Matt Sital-Singh
5
      Optimize your Catalogs


Plone Conference 2011
San Francisco
                         Matt Hamilton
                         Matt Sital-Singh
• Pre-load Catalog Queryplan
        • Balance your BTrees


Plone Conference 2011                  Matt Hamilton
San Francisco                          Matt Sital-Singh
effective date index:


                           1308585257

                           1319126057

                           1319730857

                           1319817170

                           1319817198

Plone Conference 2011                           Matt Hamilton
San Francisco                                   Matt Sital-Singh
1308585257

                 1319126057

                        1319730857

                               1319817170

                                      1319817198

Plone Conference 2011                       Matt Hamilton
San Francisco                               Matt Sital-Singh
• Used catalog multiplexer to assign ‘Answer’
            content type to dummy catalog

        • Moved messages to separate
            message_catalog

        • Monkey-patch items out of the catalog?

Plone Conference 2011                              Matt Hamilton
San Francisco                                      Matt Sital-Singh
6     If You Know Where
           Something Is - Don’t
                 Search for it

Plone Conference 2011
San Francisco
                              Matt Hamilton
                              Matt Sital-Singh
• Got into bad habit of using the catalog
            ALL the time

        • Don’t just blindly use catalog when
            unrestrictedTraverse / contentValues will
            do

        • RAM is cheap, user patience is NOT

Plone Conference 2011                               Matt Hamilton
San Francisco                                       Matt Sital-Singh
7        Use
   unrestrictedSearchResults
           if possible

Plone Conference 2011
San Francisco
                        Matt Hamilton
                        Matt Sital-Singh
• unrestrictedSearchResults does not check:
         • effective index
         • allowed_roles_and_users index
       ➡ A lot less work for the catalog to do

Plone Conference 2011                            Matt Hamilton
San Francisco                                    Matt Sital-Singh
8Is Plone the Right Tool?


Plone Conference 2011
San Francisco
                             Matt Hamilton
                             Matt Sital-Singh
• “When you have a hammer, everything
            looks like a nail...”

        • A lot of this data is relational...
        • Not much of it is CMSish
        • Maybe should have used Pyramid?

Plone Conference 2011                           Matt Hamilton
San Francisco                                   Matt Sital-Singh
9        Use Event Driven
                  Workflow
              (Don’t recalculate)

Plone Conference 2011
San Francisco
                                    Matt Hamilton
                                    Matt Sital-Singh
• User rating / Course ranking
        • Use workflow event subscribers, don’t re-
            calculate on view every time




Plone Conference 2011                          Matt Hamilton
San Francisco                                  Matt Sital-Singh
10     Some things don’t need
          to be immediate

Plone Conference 2011
San Francisco
                           Matt Hamilton
                           Matt Sital-Singh
• “You have 10 new messages”
        • Not everything has to be updated all the
            time in every location on the site




Plone Conference 2011                            Matt Hamilton
San Francisco                                    Matt Sital-Singh
Thanks!


Plone Conference 2011             Matt Hamilton
San Francisco                     Matt Sital-Singh

More Related Content

More from Matt Hamilton

More from Matt Hamilton (17)

Supercharge Your Career with Open Source
Supercharge Your Career with Open SourceSupercharge Your Career with Open Source
Supercharge Your Career with Open Source
 
Plone and Sharepoint
Plone and SharepointPlone and Sharepoint
Plone and Sharepoint
 
How to get started with the Pluggable Authentication System
How to get started with the Pluggable Authentication SystemHow to get started with the Pluggable Authentication System
How to get started with the Pluggable Authentication System
 
Plone and Single-Sign On - Active Directory and the Holy Grail
Plone and Single-Sign On - Active Directory and the Holy GrailPlone and Single-Sign On - Active Directory and the Holy Grail
Plone and Single-Sign On - Active Directory and the Holy Grail
 
BathCamp #32 - CMS Smackdown! - Plone
BathCamp #32 - CMS Smackdown! - PloneBathCamp #32 - CMS Smackdown! - Plone
BathCamp #32 - CMS Smackdown! - Plone
 
Plone Symposium East 2011 Keynote: Plone, A Solution not a Product
Plone Symposium East 2011 Keynote: Plone, A Solution not a ProductPlone Symposium East 2011 Keynote: Plone, A Solution not a Product
Plone Symposium East 2011 Keynote: Plone, A Solution not a Product
 
Plone: Event Driven Programming
Plone: Event Driven ProgrammingPlone: Event Driven Programming
Plone: Event Driven Programming
 
Mountain Tops to Archipelagos - The People Behind Plone (+AUDIO)
Mountain Tops to Archipelagos - The People Behind Plone (+AUDIO)Mountain Tops to Archipelagos - The People Behind Plone (+AUDIO)
Mountain Tops to Archipelagos - The People Behind Plone (+AUDIO)
 
The Flexibility of Open Source - Plone in the Public Sector
The Flexibility of Open Source - Plone in the Public SectorThe Flexibility of Open Source - Plone in the Public Sector
The Flexibility of Open Source - Plone in the Public Sector
 
The Flexibility of Open Source: A Case Study of a large Corporate Intranet
The Flexibility of Open Source: A Case Study of a large Corporate IntranetThe Flexibility of Open Source: A Case Study of a large Corporate Intranet
The Flexibility of Open Source: A Case Study of a large Corporate Intranet
 
Plone - Revised Roadmap: Plone 3,4,5 and beyond - Dutch Plone Users Day (+AUDIO)
Plone - Revised Roadmap: Plone 3,4,5 and beyond - Dutch Plone Users Day (+AUDIO)Plone - Revised Roadmap: Plone 3,4,5 and beyond - Dutch Plone Users Day (+AUDIO)
Plone - Revised Roadmap: Plone 3,4,5 and beyond - Dutch Plone Users Day (+AUDIO)
 
An introduction to Zope Page Templates and their use outside of Zope (+Audio)
An introduction to Zope Page Templates and their use outside of Zope (+Audio)An introduction to Zope Page Templates and their use outside of Zope (+Audio)
An introduction to Zope Page Templates and their use outside of Zope (+Audio)
 
Lipstick On a Pig (+Audio)
Lipstick On a Pig (+Audio)Lipstick On a Pig (+Audio)
Lipstick On a Pig (+Audio)
 
Lipstick on a Pig - European Plone Symposium 2009
Lipstick on a Pig - European Plone Symposium 2009Lipstick on a Pig - European Plone Symposium 2009
Lipstick on a Pig - European Plone Symposium 2009
 
Kent Connects: Harnessing Open Source for Shared Services and Partnership Wor...
Kent Connects: Harnessing Open Source for Shared Services and Partnership Wor...Kent Connects: Harnessing Open Source for Shared Services and Partnership Wor...
Kent Connects: Harnessing Open Source for Shared Services and Partnership Wor...
 
NextGen Roadshow Bmex Case Study
NextGen Roadshow Bmex Case StudyNextGen Roadshow Bmex Case Study
NextGen Roadshow Bmex Case Study
 
Open Source and Content Management (+audio)
Open Source and Content Management (+audio)Open Source and Content Management (+audio)
Open Source and Content Management (+audio)
 

Recently uploaded

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Peter Udo Diehl
 

Recently uploaded (20)

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........Bits & Pixels using AI for Good.........
Bits & Pixels using AI for Good.........
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
 
ODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User GroupODC, Data Fabric and Architecture User Group
ODC, Data Fabric and Architecture User Group
 
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
Knowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and backKnowledge engineering: from people to machines and back
Knowledge engineering: from people to machines and back
 
Speed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in MinutesSpeed Wins: From Kafka to APIs in Minutes
Speed Wins: From Kafka to APIs in Minutes
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 

Mistakes Made and Lessons Learnt Scaling Plone post-Launch

  • 1.
  • 2. Matt Hamilton & Matt Sital-Singh Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 3. 0 Plone Conference 2011 San Francisco Background Matt Hamilton Matt Sital-Singh
  • 4. Use Case • Learning Management-type system • A lot of content • Users ‘editing’ content, but not necessarily aware of it • Lots of ‘clustering’ of load on resources Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 5. Scale • Plone 3 • User accounts: 30,000 • Messages: 160,000 • Course questions: 260,000 • Other: 60,000 Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 6. 1 Don’t Create Ghost Content Plone Conference 2011 San Francisco Matt Hamilton Matt Sital-Singh
  • 7. • User perception of speed • If they expect a ‘read’ they want a ‘read’ - user signups - course starting • Will accept a delay if you are actually processing something Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 8. 2 Use Optimization Products Plone Conference 2011 San Francisco Matt Hamilton Matt Sital-Singh
  • 9. • experimental.catalogqueryplan • experimental.contentcreation • experimental.daterangeindexoptimisations • experimental.aggressiveopaquespeedup • collective.indexing Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 10. 3 One Plone site or Multiple? Plone Conference 2011 San Francisco Matt Hamilton Matt Sital-Singh
  • 11. • One Plone site per ‘site’ • Multiple ‘sites’ per Plone site • Common content? Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 12. 4 Large Catalogs are Slow Plone Conference 2011 San Francisco Matt Hamilton Matt Sital-Singh
  • 13. • We had 260,000 questions • each had 2-4 answers • = 520,000 - 1,400,000 answers All in portal_catalog! Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 14. 5 Optimize your Catalogs Plone Conference 2011 San Francisco Matt Hamilton Matt Sital-Singh
  • 15. • Pre-load Catalog Queryplan • Balance your BTrees Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 16. effective date index: 1308585257 1319126057 1319730857 1319817170 1319817198 Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 17. 1308585257 1319126057 1319730857 1319817170 1319817198 Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 18. • Used catalog multiplexer to assign ‘Answer’ content type to dummy catalog • Moved messages to separate message_catalog • Monkey-patch items out of the catalog? Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 19. 6 If You Know Where Something Is - Don’t Search for it Plone Conference 2011 San Francisco Matt Hamilton Matt Sital-Singh
  • 20. • Got into bad habit of using the catalog ALL the time • Don’t just blindly use catalog when unrestrictedTraverse / contentValues will do • RAM is cheap, user patience is NOT Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 21. 7 Use unrestrictedSearchResults if possible Plone Conference 2011 San Francisco Matt Hamilton Matt Sital-Singh
  • 22. • unrestrictedSearchResults does not check: • effective index • allowed_roles_and_users index ➡ A lot less work for the catalog to do Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 23. 8Is Plone the Right Tool? Plone Conference 2011 San Francisco Matt Hamilton Matt Sital-Singh
  • 24. • “When you have a hammer, everything looks like a nail...” • A lot of this data is relational... • Not much of it is CMSish • Maybe should have used Pyramid? Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 25. 9 Use Event Driven Workflow (Don’t recalculate) Plone Conference 2011 San Francisco Matt Hamilton Matt Sital-Singh
  • 26. • User rating / Course ranking • Use workflow event subscribers, don’t re- calculate on view every time Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 27. 10 Some things don’t need to be immediate Plone Conference 2011 San Francisco Matt Hamilton Matt Sital-Singh
  • 28. • “You have 10 new messages” • Not everything has to be updated all the time in every location on the site Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh
  • 29. Thanks! Plone Conference 2011 Matt Hamilton San Francisco Matt Sital-Singh