Plone TuneUp challenges

2,137 views
2,026 views

Published on

Published in: Technology, Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,137
On SlideShare
0
From Embeds
0
Number of Embeds
26
Actions
Shares
0
Downloads
22
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Plone TuneUp challenges

  1. 1. 1 Plone TuneUp challenges in search of enterprise intranet solution Andrew Mleczko (redturtle.net) European Plone Symposium 2009 Sorrento, Italy
  2. 2. 2 Poznan Ferrara
  3. 3. 3 Co-founder of ACC Poland Ltd., the first Polish ● Plone company (past) Plone consultant at RedTurtle Technology - ● based in Ferrara (present) Plone-based and Lotus-based web applications ●
  4. 4. 4 RedTurtle Technology “Sharing knowledge about knowledge sharing”
  5. 5. 5 So what can you expect... Plone is just a CMS? ● Case study – Intranet Provincia di Ferrara ● Loadtests environment ● Plone TuneUp challenges ●
  6. 6. 6 How we can use Plone?
  7. 7. 7 Plone variety Community portals ● Document management system (DMS) ● Groupware - Collaboration ● Subsite management system ● Enterprise intranets ● ... ● Combination of all mantioned above ●
  8. 8. 8 Plone as CMS vs. Plone as framework
  9. 9. 9 “Plone is NOT a general web development framework.“ /Carlos de la Guardia/ Why?
  10. 10. 10 Plone can be SLOW* *when using the wrong way
  11. 11. 11 Framework comparision Used hardware: ● CPU: AMD OpteronT Processor 146 (2 GHz) – Memory: 2 GB – Testing with Siege 2.68 ● Read scenario ● Results for 50, 200, 300 Concurent Users (CU) ● More details: ● www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks
  12. 12. 12 Comparision Response time, secs. 2,5 2 1,5 50 CU 200 CU 300 CU 1 0,5 0 Django RubyOnRails TurboGears
  13. 13. 13 Comparision Longest transaction, secs. 35 30 25 20 50 CU 200 CU 15 300 CU 10 5 0 Django RubyOnRails TurboGears
  14. 14. 14 So where is Plone?
  15. 15. 15 Far, far away Response time, secs. Longest transaction, secs. 18 60 16 50 14 12 40 10 50 CU 50 CU 30 200 CU 200 CU 300 CU 300 CU 8 6 20 4 10 2 0 0 Django RubyOnRails TurboGears Plone Django RubyOnRails TurboGears Plone
  16. 16. 16 But comparing with Zope2 Response time, secs. 18 16 14 12 10 50 CU 200 CU 8 300 CU 6 4 2 0 Django + Psyco RubyOnRails 1.2.1 TurboGears Plone Zope2
  17. 17. 17 What can we do... Do not treat Plone as a framework ● For some use cases: ● use ContentMirror + fast framework (django, pylons, repoze) But what about enterprise intranet solutions? ●
  18. 18. 18 Plone for enterprise intranet Large number of users (more than 500) and ● documents (more than 100 000) A lot of simultaneous write requests (>20 CU/s) ● Using of caching limited ● Slowest acceptable resp. time <10 seconds ● the limit for keeping the user's attention focused [Miller 1968; Card et al. 1991]
  19. 19. 19 Intranet Provincia di Ferrara y d tu s e s a C
  20. 20. 20 Prerequisites 800 users stored in Domino Server (by PAS) ● 170.000 documents ● Yearly growth: ca 25.000 documents ● Peak traffic: 20 CU/s - write transactions ● (based on Domino logs)
  21. 21. 21 Objective User response time (mixed read/write): ● Slowest (5-6 seconds) – Recommended (2-4 seconds) 90% of all request – Fast (<2 seconds) –
  22. 22. 22 Test environment Hardware (current): ● 4 dual-core Intel Xeon 2.8 Ghz – 32 GB RAM – Amazon EC2 (future) ●
  23. 23. 23 Test environment collective.loadtesting ● funkload – collective.funkload – PageTemplate Profiler ● mr.bent ● Munin + redturtle.munin ●
  24. 24. 24 collective.loadtesting class Readonly(testcase.FLTestCase):          def setUp(self):         self.server_url = 'http://localhost:8080/plone'     def test_ReadOnly(self):         server_url = self.server_url         self.get(server_url + quot;/plone/login_formquot;,             description=quot;Get /plone/login_formquot;)         self.post(server_url + quot;/plone/login_formquot;, params=[             ['came_from', server_url + '/plone'],             ['form.submitted', '1'],             ['cookies_enabled', ''],             ['login_name', ''],             ['pwd_empty', '0'],             ['__ac_name', 'admin'],             ['__ac_password', 'admin'],             ['submit', 'Accedi']],             description=quot;Post /plone/login_formquot;)
  25. 25. 25 mr.bent >>> from mr.bent.wrapper import mkwrapper >>> from mr.bent.tests.test_bent import callcounter >>> from Products.CMFPlone.CatalogTool import      CatalogTool >>> mkwrapper(CatalogTool.searchResults,   callcounter, quot;catalogsearchesquot;)
  26. 26. 26 Test scenario Log in ● View random folder ● Create document (2 requests) ● View newly created document ● Publish it ● View home page ● Logout ●
  27. 27. 27 Default clean Plone Response time, secs. 60 50 40 5CU 30 20CU 60CU 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  28. 28. 28 Default clean Plone % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  29. 29. 29 Let's start TuneUp and install the basics: Varnish + Pound (with sticky session) + + 6 ZEO Clients (18 threads) + ZEO Server
  30. 30. 30 Apache – proxy/rewriting urls ● Varnish – cache resources, add http headers ● Pound – balancing with sticky sessison ● 6 ZEO clients ● ZEO Server ●
  31. 31. 31 + Varnish + Pound + ZEO Response time, secs. 40 35 30 25 5CU 20 20CU 60CU 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  32. 32. 32 + Varnish + Pound + ZEO % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  33. 33. 33 Comparing to default Plone
  34. 34. 34 Comparision Response time, secs., 20 CU 40 35 30 25 20 Default Plone Improvment 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  35. 35. 35 Where are the conflict errors comming from? mainly portal_catalog
  36. 36. 36 Let's try: mount portal_catalog to different ZODB and increase zodb cache on it
  37. 37. 37 Install redturtle.catalogmount ● Add zope-conf-additional: ● <zodb_db catalog> mount-point /plone/portal_catalog container-class Products.CMFPlone.CatalogTool.CatalogTool cache-size 300000 <zeoclient> server ${zeoserver:zeo-address} storage 2 name catalogstorage var ${buildout:parts-directory}/instance1/var cache-size 400MB </zeoclient> </zodb_db>
  38. 38. 38 And add zeo-conf-additional: ● <filestorage 2> path ${buildout:directory}/var/filestorage/CatalogData.fs </filestorage>
  39. 39. 39 + redturtle.catalogmount Response time, secs. 25 20 15 5CU 20CU 60CU 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  40. 40. 40 + redturtle.catalogmount % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  41. 41. 41 Maybe there is a more stable solution: collective.indexing experimental.catalogqueryplan archetypes.schematuning
  42. 42. 42 + collective.indexing + archetypes.schematuning Response time, secs. 16 14 12 10 5CU 8 20CU 60CU 6 4 2 0 Log in Folder view Create document Save document View Publish Home page Logout
  43. 43. 43 + collective.indexing + archetypes.schematuning % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  44. 44. 44 So what is the improvment?
  45. 45. 45 Improvment Response time, secs., 20 CU 40 35 30 25 20 fresh Plone after TuneUp 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  46. 46. 46 Let's try to scale it...
  47. 47. 47 ZEO scalability problem Response time, secs. 20 CU 45 40 35 30 25 1:10 4:20 20 6:18 10:30 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  48. 48. 48 ZEO scalability problem % Errors (ConflictErrors), 20 CU 60 50 40 1:10 30 4:20 6:18 10:30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  49. 49. 49 Plone intranet limits ZODB is slow when handling concurrent writes ● transactions Scalability problem using flat ZEO ● Missing 'out of the box' performance ● (comparing to other framworks, CMS)
  50. 50. 50 How we can solve the problem?
  51. 51. 51 Ideas RelStorage for scalability problem ● collective.solr ● others ? ● share knowledge at plone-enterprise mailing list ●

×