0
1




          Plone TuneUp challenges
          in search of enterprise intranet solution


Andrew Mleczko (redturtle.ne...
2




Poznan   Ferrara
3




    Co-founder of ACC Poland Ltd., the first Polish
●



    Plone company (past)
    Plone consultant at RedTurtle ...
4



RedTurtle Technology




         “Sharing knowledge
       about knowledge sharing”
5



So what can you expect...

      Plone is just a CMS?
  ●



      Case study – Intranet Provincia di Ferrara
  ●



...
6




How we can use
    Plone?
7



Plone variety

      Community portals
  ●



      Document management system (DMS)
  ●



      Groupware - Collabo...
8




  Plone as CMS
       vs.
Plone as framework
9




“Plone is NOT a general web development
framework.“

                    /Carlos de la Guardia/



          Why?
10




Plone can be SLOW*



          *when using the wrong way
11



Framework comparision

         Used hardware:
 ●


          CPU: AMD OpteronT Processor 146 (2 GHz)
     –

      ...
12



Comparision

                  Response time, secs.
   2,5




    2




   1,5

                                   ...
13



Comparision

                 Longest transaction, secs.
   35



   30



   25



   20
                          ...
14




So where is Plone?
15



Far, far away

              Response time, secs.                            Longest transaction, secs.
18          ...
16



But comparing with Zope2

                                           Response time, secs.
 18


 16


 14


 12


 1...
17



What can we do...

      Do not treat Plone as a framework
  ●



      For some use cases:
  ●



      use Content...
18



Plone for enterprise intranet

      Large number of users (more than 500) and
  ●



      documents (more than 100...
19



Intranet Provincia di Ferrara



                                 y
                               d
               ...
20



Prerequisites

      800 users stored in Domino Server (by PAS)
  ●



      170.000 documents
  ●



      Yearly g...
21



Objective

          User response time (mixed read/write):
  ●


          Slowest (5-6 seconds)
      –

         ...
22



Test environment

          Hardware (current):
  ●


          4 dual-core Intel Xeon 2.8 Ghz
      –

          32...
23



Test environment

          collective.loadtesting
  ●


          funkload
      –

          collective.funkload
 ...
24



collective.loadtesting

   class Readonly(testcase.FLTestCase):
       
       def setUp(self):
           self.serv...
25



mr.bent


 >>> from mr.bent.wrapper import mkwrapper
 >>> from mr.bent.tests.test_bent import callcounter
 >>> from ...
26



Test scenario

      Log in
  ●



      View random folder
  ●



      Create document (2 requests)
  ●



      V...
27



Default clean Plone

                                        Response time, secs.
   60




   50




   40




    ...
28



Default clean Plone

                                         % Errors (ConflictErrors)
 100


 90


 80


 70


 60...
29




Let's start TuneUp and install
            the basics:

 Varnish + Pound (with sticky session) +
+ 6 ZEO Clients (1...
30




    Apache – proxy/rewriting urls
●



    Varnish – cache resources, add http headers
●



    Pound – balancing w...
31



+ Varnish + Pound + ZEO

                                         Response time, secs.
 40



 35



 30



 25



 ...
32



+ Varnish + Pound + ZEO

                                         % Errors (ConflictErrors)
 100


 90


 80


 70

...
33




Comparing to default Plone
34



Comparision

                                    Response time, secs., 20 CU
 40



 35



 30



 25



 20        ...
35




Where are the conflict errors
      comming from?
      mainly portal_catalog
36




             Let's try:
mount portal_catalog to different ZODB
    and increase zodb cache on it
37




    Install redturtle.catalogmount
●

    Add zope-conf-additional:
●

      <zodb_db catalog>
        mount-point ...
38




    And add zeo-conf-additional:
●


    <filestorage 2>
       path ${buildout:directory}/var/filestorage/CatalogD...
39



+ redturtle.catalogmount


                                            Response time, secs.
 25




 20




 15


  ...
40



+ redturtle.catalogmount


                                         % Errors (ConflictErrors)
 100


  90


  80


 ...
41




Maybe there is a more
    stable solution:
     collective.indexing
experimental.catalogqueryplan
  archetypes.sche...
42



+ collective.indexing + archetypes.schematuning


                                             Response time, secs.
...
43



+ collective.indexing + archetypes.schematuning


                                          % Errors (ConflictErrors...
44




So what is the improvment?
45



Improvment


                              Response time, secs., 20 CU
  40



  35



  30



  25



  20         ...
46




Let's try to scale it...
47



ZEO scalability problem

                                  Response time, secs. 20 CU
   45



   40



   35



   ...
48



ZEO scalability problem

                                   % Errors (ConflictErrors), 20 CU
   60




   50




   ...
49



Plone intranet limits

      ZODB is slow when handling concurrent writes
  ●



      transactions
      Scalabilit...
50




How we can solve the problem?
51



Ideas


      RelStorage for scalability problem
  ●



      collective.solr
  ●



      others ?
  ●



      sha...
Upcoming SlideShare
Loading in...5
×

Plone TuneUp challenges

1,829

Published on

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,829
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
22
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Plone TuneUp challenges"

  1. 1. 1 Plone TuneUp challenges in search of enterprise intranet solution Andrew Mleczko (redturtle.net) European Plone Symposium 2009 Sorrento, Italy
  2. 2. 2 Poznan Ferrara
  3. 3. 3 Co-founder of ACC Poland Ltd., the first Polish ● Plone company (past) Plone consultant at RedTurtle Technology - ● based in Ferrara (present) Plone-based and Lotus-based web applications ●
  4. 4. 4 RedTurtle Technology “Sharing knowledge about knowledge sharing”
  5. 5. 5 So what can you expect... Plone is just a CMS? ● Case study – Intranet Provincia di Ferrara ● Loadtests environment ● Plone TuneUp challenges ●
  6. 6. 6 How we can use Plone?
  7. 7. 7 Plone variety Community portals ● Document management system (DMS) ● Groupware - Collaboration ● Subsite management system ● Enterprise intranets ● ... ● Combination of all mantioned above ●
  8. 8. 8 Plone as CMS vs. Plone as framework
  9. 9. 9 “Plone is NOT a general web development framework.“ /Carlos de la Guardia/ Why?
  10. 10. 10 Plone can be SLOW* *when using the wrong way
  11. 11. 11 Framework comparision Used hardware: ● CPU: AMD OpteronT Processor 146 (2 GHz) – Memory: 2 GB – Testing with Siege 2.68 ● Read scenario ● Results for 50, 200, 300 Concurent Users (CU) ● More details: ● www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks
  12. 12. 12 Comparision Response time, secs. 2,5 2 1,5 50 CU 200 CU 300 CU 1 0,5 0 Django RubyOnRails TurboGears
  13. 13. 13 Comparision Longest transaction, secs. 35 30 25 20 50 CU 200 CU 15 300 CU 10 5 0 Django RubyOnRails TurboGears
  14. 14. 14 So where is Plone?
  15. 15. 15 Far, far away Response time, secs. Longest transaction, secs. 18 60 16 50 14 12 40 10 50 CU 50 CU 30 200 CU 200 CU 300 CU 300 CU 8 6 20 4 10 2 0 0 Django RubyOnRails TurboGears Plone Django RubyOnRails TurboGears Plone
  16. 16. 16 But comparing with Zope2 Response time, secs. 18 16 14 12 10 50 CU 200 CU 8 300 CU 6 4 2 0 Django + Psyco RubyOnRails 1.2.1 TurboGears Plone Zope2
  17. 17. 17 What can we do... Do not treat Plone as a framework ● For some use cases: ● use ContentMirror + fast framework (django, pylons, repoze) But what about enterprise intranet solutions? ●
  18. 18. 18 Plone for enterprise intranet Large number of users (more than 500) and ● documents (more than 100 000) A lot of simultaneous write requests (>20 CU/s) ● Using of caching limited ● Slowest acceptable resp. time <10 seconds ● the limit for keeping the user's attention focused [Miller 1968; Card et al. 1991]
  19. 19. 19 Intranet Provincia di Ferrara y d tu s e s a C
  20. 20. 20 Prerequisites 800 users stored in Domino Server (by PAS) ● 170.000 documents ● Yearly growth: ca 25.000 documents ● Peak traffic: 20 CU/s - write transactions ● (based on Domino logs)
  21. 21. 21 Objective User response time (mixed read/write): ● Slowest (5-6 seconds) – Recommended (2-4 seconds) 90% of all request – Fast (<2 seconds) –
  22. 22. 22 Test environment Hardware (current): ● 4 dual-core Intel Xeon 2.8 Ghz – 32 GB RAM – Amazon EC2 (future) ●
  23. 23. 23 Test environment collective.loadtesting ● funkload – collective.funkload – PageTemplate Profiler ● mr.bent ● Munin + redturtle.munin ●
  24. 24. 24 collective.loadtesting class Readonly(testcase.FLTestCase):          def setUp(self):         self.server_url = 'http://localhost:8080/plone'     def test_ReadOnly(self):         server_url = self.server_url         self.get(server_url + quot;/plone/login_formquot;,             description=quot;Get /plone/login_formquot;)         self.post(server_url + quot;/plone/login_formquot;, params=[             ['came_from', server_url + '/plone'],             ['form.submitted', '1'],             ['cookies_enabled', ''],             ['login_name', ''],             ['pwd_empty', '0'],             ['__ac_name', 'admin'],             ['__ac_password', 'admin'],             ['submit', 'Accedi']],             description=quot;Post /plone/login_formquot;)
  25. 25. 25 mr.bent >>> from mr.bent.wrapper import mkwrapper >>> from mr.bent.tests.test_bent import callcounter >>> from Products.CMFPlone.CatalogTool import      CatalogTool >>> mkwrapper(CatalogTool.searchResults,   callcounter, quot;catalogsearchesquot;)
  26. 26. 26 Test scenario Log in ● View random folder ● Create document (2 requests) ● View newly created document ● Publish it ● View home page ● Logout ●
  27. 27. 27 Default clean Plone Response time, secs. 60 50 40 5CU 30 20CU 60CU 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  28. 28. 28 Default clean Plone % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  29. 29. 29 Let's start TuneUp and install the basics: Varnish + Pound (with sticky session) + + 6 ZEO Clients (18 threads) + ZEO Server
  30. 30. 30 Apache – proxy/rewriting urls ● Varnish – cache resources, add http headers ● Pound – balancing with sticky sessison ● 6 ZEO clients ● ZEO Server ●
  31. 31. 31 + Varnish + Pound + ZEO Response time, secs. 40 35 30 25 5CU 20 20CU 60CU 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  32. 32. 32 + Varnish + Pound + ZEO % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  33. 33. 33 Comparing to default Plone
  34. 34. 34 Comparision Response time, secs., 20 CU 40 35 30 25 20 Default Plone Improvment 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  35. 35. 35 Where are the conflict errors comming from? mainly portal_catalog
  36. 36. 36 Let's try: mount portal_catalog to different ZODB and increase zodb cache on it
  37. 37. 37 Install redturtle.catalogmount ● Add zope-conf-additional: ● <zodb_db catalog> mount-point /plone/portal_catalog container-class Products.CMFPlone.CatalogTool.CatalogTool cache-size 300000 <zeoclient> server ${zeoserver:zeo-address} storage 2 name catalogstorage var ${buildout:parts-directory}/instance1/var cache-size 400MB </zeoclient> </zodb_db>
  38. 38. 38 And add zeo-conf-additional: ● <filestorage 2> path ${buildout:directory}/var/filestorage/CatalogData.fs </filestorage>
  39. 39. 39 + redturtle.catalogmount Response time, secs. 25 20 15 5CU 20CU 60CU 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  40. 40. 40 + redturtle.catalogmount % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  41. 41. 41 Maybe there is a more stable solution: collective.indexing experimental.catalogqueryplan archetypes.schematuning
  42. 42. 42 + collective.indexing + archetypes.schematuning Response time, secs. 16 14 12 10 5CU 8 20CU 60CU 6 4 2 0 Log in Folder view Create document Save document View Publish Home page Logout
  43. 43. 43 + collective.indexing + archetypes.schematuning % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  44. 44. 44 So what is the improvment?
  45. 45. 45 Improvment Response time, secs., 20 CU 40 35 30 25 20 fresh Plone after TuneUp 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  46. 46. 46 Let's try to scale it...
  47. 47. 47 ZEO scalability problem Response time, secs. 20 CU 45 40 35 30 25 1:10 4:20 20 6:18 10:30 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  48. 48. 48 ZEO scalability problem % Errors (ConflictErrors), 20 CU 60 50 40 1:10 30 4:20 6:18 10:30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  49. 49. 49 Plone intranet limits ZODB is slow when handling concurrent writes ● transactions Scalability problem using flat ZEO ● Missing 'out of the box' performance ● (comparing to other framworks, CMS)
  50. 50. 50 How we can solve the problem?
  51. 51. 51 Ideas RelStorage for scalability problem ● collective.solr ● others ? ● share knowledge at plone-enterprise mailing list ●
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×