Plone TuneUp challenges

  • 1,721 views
Uploaded on

 

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,721
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
22
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. 1 Plone TuneUp challenges in search of enterprise intranet solution Andrew Mleczko (redturtle.net) European Plone Symposium 2009 Sorrento, Italy
  • 2. 2 Poznan Ferrara
  • 3. 3 Co-founder of ACC Poland Ltd., the first Polish ● Plone company (past) Plone consultant at RedTurtle Technology - ● based in Ferrara (present) Plone-based and Lotus-based web applications ●
  • 4. 4 RedTurtle Technology “Sharing knowledge about knowledge sharing”
  • 5. 5 So what can you expect... Plone is just a CMS? ● Case study – Intranet Provincia di Ferrara ● Loadtests environment ● Plone TuneUp challenges ●
  • 6. 6 How we can use Plone?
  • 7. 7 Plone variety Community portals ● Document management system (DMS) ● Groupware - Collaboration ● Subsite management system ● Enterprise intranets ● ... ● Combination of all mantioned above ●
  • 8. 8 Plone as CMS vs. Plone as framework
  • 9. 9 “Plone is NOT a general web development framework.“ /Carlos de la Guardia/ Why?
  • 10. 10 Plone can be SLOW* *when using the wrong way
  • 11. 11 Framework comparision Used hardware: ● CPU: AMD OpteronT Processor 146 (2 GHz) – Memory: 2 GB – Testing with Siege 2.68 ● Read scenario ● Results for 50, 200, 300 Concurent Users (CU) ● More details: ● www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks
  • 12. 12 Comparision Response time, secs. 2,5 2 1,5 50 CU 200 CU 300 CU 1 0,5 0 Django RubyOnRails TurboGears
  • 13. 13 Comparision Longest transaction, secs. 35 30 25 20 50 CU 200 CU 15 300 CU 10 5 0 Django RubyOnRails TurboGears
  • 14. 14 So where is Plone?
  • 15. 15 Far, far away Response time, secs. Longest transaction, secs. 18 60 16 50 14 12 40 10 50 CU 50 CU 30 200 CU 200 CU 300 CU 300 CU 8 6 20 4 10 2 0 0 Django RubyOnRails TurboGears Plone Django RubyOnRails TurboGears Plone
  • 16. 16 But comparing with Zope2 Response time, secs. 18 16 14 12 10 50 CU 200 CU 8 300 CU 6 4 2 0 Django + Psyco RubyOnRails 1.2.1 TurboGears Plone Zope2
  • 17. 17 What can we do... Do not treat Plone as a framework ● For some use cases: ● use ContentMirror + fast framework (django, pylons, repoze) But what about enterprise intranet solutions? ●
  • 18. 18 Plone for enterprise intranet Large number of users (more than 500) and ● documents (more than 100 000) A lot of simultaneous write requests (>20 CU/s) ● Using of caching limited ● Slowest acceptable resp. time <10 seconds ● the limit for keeping the user's attention focused [Miller 1968; Card et al. 1991]
  • 19. 19 Intranet Provincia di Ferrara y d tu s e s a C
  • 20. 20 Prerequisites 800 users stored in Domino Server (by PAS) ● 170.000 documents ● Yearly growth: ca 25.000 documents ● Peak traffic: 20 CU/s - write transactions ● (based on Domino logs)
  • 21. 21 Objective User response time (mixed read/write): ● Slowest (5-6 seconds) – Recommended (2-4 seconds) 90% of all request – Fast (<2 seconds) –
  • 22. 22 Test environment Hardware (current): ● 4 dual-core Intel Xeon 2.8 Ghz – 32 GB RAM – Amazon EC2 (future) ●
  • 23. 23 Test environment collective.loadtesting ● funkload – collective.funkload – PageTemplate Profiler ● mr.bent ● Munin + redturtle.munin ●
  • 24. 24 collective.loadtesting class Readonly(testcase.FLTestCase):          def setUp(self):         self.server_url = 'http://localhost:8080/plone'     def test_ReadOnly(self):         server_url = self.server_url         self.get(server_url + quot;/plone/login_formquot;,             description=quot;Get /plone/login_formquot;)         self.post(server_url + quot;/plone/login_formquot;, params=[             ['came_from', server_url + '/plone'],             ['form.submitted', '1'],             ['cookies_enabled', ''],             ['login_name', ''],             ['pwd_empty', '0'],             ['__ac_name', 'admin'],             ['__ac_password', 'admin'],             ['submit', 'Accedi']],             description=quot;Post /plone/login_formquot;)
  • 25. 25 mr.bent >>> from mr.bent.wrapper import mkwrapper >>> from mr.bent.tests.test_bent import callcounter >>> from Products.CMFPlone.CatalogTool import      CatalogTool >>> mkwrapper(CatalogTool.searchResults,   callcounter, quot;catalogsearchesquot;)
  • 26. 26 Test scenario Log in ● View random folder ● Create document (2 requests) ● View newly created document ● Publish it ● View home page ● Logout ●
  • 27. 27 Default clean Plone Response time, secs. 60 50 40 5CU 30 20CU 60CU 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 28. 28 Default clean Plone % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 29. 29 Let's start TuneUp and install the basics: Varnish + Pound (with sticky session) + + 6 ZEO Clients (18 threads) + ZEO Server
  • 30. 30 Apache – proxy/rewriting urls ● Varnish – cache resources, add http headers ● Pound – balancing with sticky sessison ● 6 ZEO clients ● ZEO Server ●
  • 31. 31 + Varnish + Pound + ZEO Response time, secs. 40 35 30 25 5CU 20 20CU 60CU 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 32. 32 + Varnish + Pound + ZEO % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 33. 33 Comparing to default Plone
  • 34. 34 Comparision Response time, secs., 20 CU 40 35 30 25 20 Default Plone Improvment 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 35. 35 Where are the conflict errors comming from? mainly portal_catalog
  • 36. 36 Let's try: mount portal_catalog to different ZODB and increase zodb cache on it
  • 37. 37 Install redturtle.catalogmount ● Add zope-conf-additional: ● <zodb_db catalog> mount-point /plone/portal_catalog container-class Products.CMFPlone.CatalogTool.CatalogTool cache-size 300000 <zeoclient> server ${zeoserver:zeo-address} storage 2 name catalogstorage var ${buildout:parts-directory}/instance1/var cache-size 400MB </zeoclient> </zodb_db>
  • 38. 38 And add zeo-conf-additional: ● <filestorage 2> path ${buildout:directory}/var/filestorage/CatalogData.fs </filestorage>
  • 39. 39 + redturtle.catalogmount Response time, secs. 25 20 15 5CU 20CU 60CU 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 40. 40 + redturtle.catalogmount % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 41. 41 Maybe there is a more stable solution: collective.indexing experimental.catalogqueryplan archetypes.schematuning
  • 42. 42 + collective.indexing + archetypes.schematuning Response time, secs. 16 14 12 10 5CU 8 20CU 60CU 6 4 2 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 43. 43 + collective.indexing + archetypes.schematuning % Errors (ConflictErrors) 100 90 80 70 60 5CU 50 20CU 60CU 40 30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 44. 44 So what is the improvment?
  • 45. 45 Improvment Response time, secs., 20 CU 40 35 30 25 20 fresh Plone after TuneUp 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 46. 46 Let's try to scale it...
  • 47. 47 ZEO scalability problem Response time, secs. 20 CU 45 40 35 30 25 1:10 4:20 20 6:18 10:30 15 10 5 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 48. 48 ZEO scalability problem % Errors (ConflictErrors), 20 CU 60 50 40 1:10 30 4:20 6:18 10:30 20 10 0 Log in Folder view Create document Save document View Publish Home page Logout
  • 49. 49 Plone intranet limits ZODB is slow when handling concurrent writes ● transactions Scalability problem using flat ZEO ● Missing 'out of the box' performance ● (comparing to other framworks, CMS)
  • 50. 50 How we can solve the problem?
  • 51. 51 Ideas RelStorage for scalability problem ● collective.solr ● others ? ● share knowledge at plone-enterprise mailing list ●