SlideShare a Scribd company logo
1 of 27
Download to read offline
unloading plone
Tools, Tips, and Techniques for Approaching Scalability


Elizabeth Leddy
@eleddy
am I in the right room?
                               Symptoms Include
 inability to grow with traditional caching solutions,
     icky system performance, crashitis, log-o-phobia,
          hardware anemia, mashup mayhem, diminishing
         confidence in your abilities as plone integrator or
 administrator, where do I start migraines, all the rest of the
  talks look boring, this room has better internet connectivity, I
                                have no clue what I’m doing here
CONFUCIUS
                               SAY



A responsive, scalable Plone setup has little to do with
optimizing code
Measure, Measure, Measure
 Measure, Measure, Measure
   Measure, Measure, Measure
     Measure, Measure, Measure
      Measure, Measure, Measure
        Measure, Measure, Measure
          Measure, Measure, Measure
             Measure, Measure, Measure
               Measure, Measure, Measure

                       Act
AUTO-RESTART
GANGLIA ZOPE HEALTH WATCHER

MUNIN MONIT CUSTOM PLUGINS TIMEOUTS
ALERTS                TIME BASED GRAPHS   AFFECTION


                              ABNORMALITIES


                                   I quit...




             Monitoring Tools
Watching Bad Performances
Whale watchers are
triggered by system
abnormalities, and
grab relevant
information to help
you diagnose what
happened.
Simple(r) Error Monitoring
                   Sludge Through Logs

                    Escalate with Email

                   Google Analytics for
                 Timeouts, 4XX & 5XX
4 Stages of System Grief
Denial     Anger     Bargaining   Acceptance
                                     LB
HTTPD     HTTPD       HTTPD       HTTPD^N

                      CACHE        CACHES

          PROXY       PROXY        PROXY

ZOPE      ZOPE++     ZOPE++++      ZOPE^N

                     DB CACHE     DB CACHES
                                     LB
ZEO      ZEO SHARD   ZEO SHARD    ZEO & CO.
H OW MUCH
A      CPU    DISK
R     HAPROXY
D POUND          ZEO
W    HTTPD          SQUID
A   NGINX
R            ZOPE
                             VARNISH


E                           ZEO CACHE?


             MEMCACHE   ZODB CACHE


                    RAM
Two’s Company, Three’s a Crowd




ZOPE++
DB CACHE
How many zopes should I have?
                          50% avg util
                          2 zopes/cpu
                   api/async instances?




ZOPE++
Complex by Default
The CacheFu
                   Conundrum
The browser is the cheapest, easiest caching tool
at your disposal, Daniel-son. It’s time to show the
world you are the developer I know you can be...



                 Oh, CacheFu you have been a great
              leader and I wish I would have taken the
                time to figure out how caching really
              works. I’m afraid I will fail you, never fully
                   understanding the 304. Shame!
Serve STATIC content from a STATIC web server
to unload plone, provide faster
response time to initial requests,                                        5200

and enable simpler caching
strategies


                                                                 2700
                                                    2600

                            2100
                    1800
                                          1500



        httpd
        plone
                      user.gif             link_icon.gif            poo.gif



HTTPD     RewriteRule ^/(.*)portal_skins/mysite_images/(.*) /mysite_images/$2 [L]
CSS & Sprites
30 requests, ~200 kb
.contenttype-folder{
   background-image: url(folder.gif);
}




1 request, ~18 kb
.contenttype-folder{
   background-image: url(all-my-images.png);
   background-position: 75px 25px;
}




                                               UNLOAD

HTTPD
Picking a Proxy or:
 How I learned to stop worrying and love HAProxy

   graceful reloads

   backend health

   distribution algorithms

   warmup time

   preserve keepalives

   web based stats




PROXY
Playing Nice with Others
 if python < 2.6:
     socket.settimeout(2)
 else:
     urllib2.urlopen(url, timeout=2)




ZOPE++

ZEO & CO.
Pickle
                                Drop
                               Pickup
                              Process
ZOPE++                        Callback
         Pickle Me Some Asynchronicity
EXTERNAL DATA ST.




ZEO & CO.
                      tal:on-error
ACCEPTANCE
                 ZEO is perfect for object
              oriented access, but it needs
               a little help with other data
                           access paradigms




ZEO & CO.
RESTARTing zope is NOT a SIN
                   sometimes its just easier...

            a   PACKED zodb is a HAPPY zodb
                           beware of backup side effects
                 CRON is
                    deploy with 1-step
                                       HOMEBOY
                           fabrics’ rolling blackouts


 FACTS
ZOPE
ZEO & CO.
ON BACKUPS
                     use repozo

                  chunked rsync

                  watch the disk




ZEO & CO.
Plone != Drupal   Scalability = Talent + Hardware
Living in the Cloud
          The Oath:
           I will automate a modular
           installation of everything
           I will not do anything manually
           that could be done
           automatically, which is everything
           I can trigger unscheduled
           processes anytime, from
           anywhere
           I appreciate that scaling is hard:
           the cloud offers options, not
           solutions
‣   Always setup system and error monitoring first
‣   Choose the right software for YOUR hardware
‣   Use as many zopes as possible but no more
‣   Do it like a flickr engineer
‣   Don’t make zope handle unnecessary requests
‣   Never underestimate the importance of a
    proper disk-RAM partnership
‣   Design processes to be able to run from
    anywhere, at anytime, on any machine




               SUMMARY
All this and more at
http://scalingplone.pbwiki.com




              Thank You!

More Related Content

Similar to Unloading Plone

Prophet - Beijing Perl Workshop
Prophet - Beijing Perl WorkshopProphet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
Jesse Vincent
 
Cloud Best Practices
Cloud Best PracticesCloud Best Practices
Cloud Best Practices
Eric Bottard
 
Site Performance - From Pinto to Ferrari
Site Performance - From Pinto to FerrariSite Performance - From Pinto to Ferrari
Site Performance - From Pinto to Ferrari
Joseph Scott
 
Puppet for dummies - PHPBenelux UG edition
Puppet for dummies - PHPBenelux UG editionPuppet for dummies - PHPBenelux UG edition
Puppet for dummies - PHPBenelux UG edition
Joshua Thijssen
 
Pse2010 rel storage
Pse2010 rel storagePse2010 rel storage
Pse2010 rel storage
Lars Noldan
 

Similar to Unloading Plone (20)

Docker all the way
Docker all the wayDocker all the way
Docker all the way
 
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl WorkshopProphet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
 
Corwin on Containers
Corwin on ContainersCorwin on Containers
Corwin on Containers
 
Cfgmgmt Challenges aren't technical anymore
Cfgmgmt Challenges aren't technical anymoreCfgmgmt Challenges aren't technical anymore
Cfgmgmt Challenges aren't technical anymore
 
Cloud Computing Bootcamp On The Google App Engine [v1.1]
Cloud Computing Bootcamp On The Google App Engine [v1.1]Cloud Computing Bootcamp On The Google App Engine [v1.1]
Cloud Computing Bootcamp On The Google App Engine [v1.1]
 
Racing with Droids
Racing with DroidsRacing with Droids
Racing with Droids
 
ZODB Tips and Tricks
ZODB Tips and TricksZODB Tips and Tricks
ZODB Tips and Tricks
 
ZODB Tips and Tricks
ZODB Tips and TricksZODB Tips and Tricks
ZODB Tips and Tricks
 
Cloud Best Practices
Cloud Best PracticesCloud Best Practices
Cloud Best Practices
 
Enhance system transparency and truthfulness with request tracing
Enhance system transparency and truthfulness with request tracingEnhance system transparency and truthfulness with request tracing
Enhance system transparency and truthfulness with request tracing
 
Docker experience @inbotapp
Docker experience @inbotappDocker experience @inbotapp
Docker experience @inbotapp
 
HTTP cache @ PUG Rome 03-29-2011
HTTP cache @ PUG Rome 03-29-2011HTTP cache @ PUG Rome 03-29-2011
HTTP cache @ PUG Rome 03-29-2011
 
Site Performance - From Pinto to Ferrari
Site Performance - From Pinto to FerrariSite Performance - From Pinto to Ferrari
Site Performance - From Pinto to Ferrari
 
Puppet for dummies - PHPBenelux UG edition
Puppet for dummies - PHPBenelux UG editionPuppet for dummies - PHPBenelux UG edition
Puppet for dummies - PHPBenelux UG edition
 
Neo4j Stored Procedure Training Part 1
Neo4j Stored Procedure Training Part 1Neo4j Stored Procedure Training Part 1
Neo4j Stored Procedure Training Part 1
 
Austin Web Architecture
Austin Web ArchitectureAustin Web Architecture
Austin Web Architecture
 
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASEBATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
BATTLEFIELD ORM: TIPS, TACTICS AND STRATEGIES FOR CONQUERING YOUR DATABASE
 
Pse2010 rel storage
Pse2010 rel storagePse2010 rel storage
Pse2010 rel storage
 
AI&BigData Lab. Александр Конопко "Celos: оркестрирование и тестирование зада...
AI&BigData Lab. Александр Конопко "Celos: оркестрирование и тестирование зада...AI&BigData Lab. Александр Конопко "Celos: оркестрирование и тестирование зада...
AI&BigData Lab. Александр Конопко "Celos: оркестрирование и тестирование зада...
 
Puppet for Sys Admins
Puppet for Sys AdminsPuppet for Sys Admins
Puppet for Sys Admins
 

More from Elizabeth Leddy

10 secrets to sustainable open source communities
10 secrets to sustainable open source communities10 secrets to sustainable open source communities
10 secrets to sustainable open source communities
Elizabeth Leddy
 

More from Elizabeth Leddy (8)

10 secrets to sustainable open source communities
10 secrets to sustainable open source communities10 secrets to sustainable open source communities
10 secrets to sustainable open source communities
 
A Small Group of Thoughtful, Committed [plone] People
A Small Group of Thoughtful, Committed [plone] PeopleA Small Group of Thoughtful, Committed [plone] People
A Small Group of Thoughtful, Committed [plone] People
 
I Wish I knew How to Quit You
I Wish I knew How to Quit YouI Wish I knew How to Quit You
I Wish I knew How to Quit You
 
Plone as a Development Platform
Plone as a Development PlatformPlone as a Development Platform
Plone as a Development Platform
 
Fuck youbackground
Fuck youbackgroundFuck youbackground
Fuck youbackground
 
F*ck you, Do Something
F*ck you, Do SomethingF*ck you, Do Something
F*ck you, Do Something
 
Old Dogs and New Tricks
Old Dogs and New TricksOld Dogs and New Tricks
Old Dogs and New Tricks
 
2011: Year of the fish
2011: Year of the fish2011: Year of the fish
2011: Year of the fish
 

Recently uploaded

Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
UK Journal
 

Recently uploaded (20)

TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptxBT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
BT & Neo4j _ How Knowledge Graphs help BT deliver Digital Transformation.pptx
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 

Unloading Plone

  • 1. unloading plone Tools, Tips, and Techniques for Approaching Scalability Elizabeth Leddy @eleddy
  • 2. am I in the right room? Symptoms Include inability to grow with traditional caching solutions, icky system performance, crashitis, log-o-phobia, hardware anemia, mashup mayhem, diminishing confidence in your abilities as plone integrator or administrator, where do I start migraines, all the rest of the talks look boring, this room has better internet connectivity, I have no clue what I’m doing here
  • 3. CONFUCIUS SAY A responsive, scalable Plone setup has little to do with optimizing code
  • 4.
  • 5. Measure, Measure, Measure Measure, Measure, Measure Measure, Measure, Measure Measure, Measure, Measure Measure, Measure, Measure Measure, Measure, Measure Measure, Measure, Measure Measure, Measure, Measure Measure, Measure, Measure Act
  • 6. AUTO-RESTART GANGLIA ZOPE HEALTH WATCHER MUNIN MONIT CUSTOM PLUGINS TIMEOUTS ALERTS TIME BASED GRAPHS AFFECTION ABNORMALITIES I quit... Monitoring Tools
  • 7. Watching Bad Performances Whale watchers are triggered by system abnormalities, and grab relevant information to help you diagnose what happened.
  • 8. Simple(r) Error Monitoring Sludge Through Logs Escalate with Email Google Analytics for Timeouts, 4XX & 5XX
  • 9. 4 Stages of System Grief Denial Anger Bargaining Acceptance LB HTTPD HTTPD HTTPD HTTPD^N CACHE CACHES PROXY PROXY PROXY ZOPE ZOPE++ ZOPE++++ ZOPE^N DB CACHE DB CACHES LB ZEO ZEO SHARD ZEO SHARD ZEO & CO.
  • 10. H OW MUCH A CPU DISK R HAPROXY D POUND ZEO W HTTPD SQUID A NGINX R ZOPE VARNISH E ZEO CACHE? MEMCACHE ZODB CACHE RAM
  • 11. Two’s Company, Three’s a Crowd ZOPE++ DB CACHE
  • 12. How many zopes should I have? 50% avg util 2 zopes/cpu api/async instances? ZOPE++
  • 14. The CacheFu Conundrum The browser is the cheapest, easiest caching tool at your disposal, Daniel-son. It’s time to show the world you are the developer I know you can be... Oh, CacheFu you have been a great leader and I wish I would have taken the time to figure out how caching really works. I’m afraid I will fail you, never fully understanding the 304. Shame!
  • 15. Serve STATIC content from a STATIC web server to unload plone, provide faster response time to initial requests, 5200 and enable simpler caching strategies 2700 2600 2100 1800 1500 httpd plone user.gif link_icon.gif poo.gif HTTPD RewriteRule ^/(.*)portal_skins/mysite_images/(.*) /mysite_images/$2 [L]
  • 16. CSS & Sprites 30 requests, ~200 kb .contenttype-folder{ background-image: url(folder.gif); } 1 request, ~18 kb .contenttype-folder{ background-image: url(all-my-images.png); background-position: 75px 25px; } UNLOAD HTTPD
  • 17. Picking a Proxy or: How I learned to stop worrying and love HAProxy graceful reloads backend health distribution algorithms warmup time preserve keepalives web based stats PROXY
  • 18. Playing Nice with Others if python < 2.6: socket.settimeout(2) else: urllib2.urlopen(url, timeout=2) ZOPE++ ZEO & CO.
  • 19. Pickle Drop Pickup Process ZOPE++ Callback Pickle Me Some Asynchronicity
  • 20. EXTERNAL DATA ST. ZEO & CO. tal:on-error
  • 21. ACCEPTANCE ZEO is perfect for object oriented access, but it needs a little help with other data access paradigms ZEO & CO.
  • 22. RESTARTing zope is NOT a SIN sometimes its just easier... a PACKED zodb is a HAPPY zodb beware of backup side effects CRON is deploy with 1-step HOMEBOY fabrics’ rolling blackouts FACTS ZOPE ZEO & CO.
  • 23. ON BACKUPS use repozo chunked rsync watch the disk ZEO & CO.
  • 24. Plone != Drupal Scalability = Talent + Hardware
  • 25. Living in the Cloud The Oath: I will automate a modular installation of everything I will not do anything manually that could be done automatically, which is everything I can trigger unscheduled processes anytime, from anywhere I appreciate that scaling is hard: the cloud offers options, not solutions
  • 26. Always setup system and error monitoring first ‣ Choose the right software for YOUR hardware ‣ Use as many zopes as possible but no more ‣ Do it like a flickr engineer ‣ Don’t make zope handle unnecessary requests ‣ Never underestimate the importance of a proper disk-RAM partnership ‣ Design processes to be able to run from anywhere, at anytime, on any machine SUMMARY
  • 27. All this and more at http://scalingplone.pbwiki.com Thank You!