Gael Le Mignot   How To Minimize Cpu And Memory Usage Of Zope And Plone Applications
Upcoming SlideShare
Loading in...5
×
 

Gael Le Mignot How To Minimize Cpu And Memory Usage Of Zope And Plone Applications

on

  • 6,365 views

This conference will present various methods to detect which parts of an application requires optimisation, and how to correct them. It'll cover both CPU and memory optimisation, and use different ...

This conference will present various methods to detect which parts of an application requires optimisation, and how to correct them. It'll cover both CPU and memory optimisation, and use different angles of attack, to cover various cases, and give an overview of what kind of code can waste CPU or memory in a Zope/Plone context. Unix tools will be used heavily, so the watcher is advised to have at least some basic knowledge of Unix command line.

Statistics

Views

Total Views
6,365
Views on SlideShare
6,354
Embed Views
11

Actions

Likes
0
Downloads
150
Comments
0

1 Embed 11

http://www.slideshare.net 11

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Gael Le Mignot   How To Minimize Cpu And Memory Usage Of Zope And Plone Applications Gael Le Mignot How To Minimize Cpu And Memory Usage Of Zope And Plone Applications Presentation Transcript

  • The golden rules Locating CPU usage CPU optimization Memory optimization Massive sites deployment How to minimize CPU and memory usage of Zope and Plone applications Gaël LE MIGNOT – Pilot Systems October 2007 Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage CPU optimization Memory optimization Massive sites deployment Plan Use catalog and product The golden rules 1 code Use efficient algorithm Using caches Use C code The hidden devil: Do things once conflict errors The 90-10 or 80-20 law Memory optimization 4 Locating CPU usage 2 Locating memory usage Using decorators Optimizing memory Issues and tips about benchmarks Massive sites deployment 5 Using ZEO pro and cons DeadlockDebugger Squid caching overview CPU optimization Site exporting strategies 3 Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage CPU optimization Memory optimization Massive sites deployment Why this talk ? The current situation Plone is very powerful But Plone is slow ! High-level applications developers tend to forget about optimization The goals of this talk Remind you the golden rules Provide tips, tricks and way to write faster Plone code Consider both CPU and memory usage A quick coverage of deployment solutions, the work-around Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Use efficient algorithm Locating CPU usage Use C code CPU optimization Do things once Memory optimization The 90-10 or 80-20 law Massive sites deployment Oh, no, not maths ! Algorithm is the key Efficient algorithm is the key to fast programs The gains for efficient algorithms are in powers of ten It is not obsoleted by high-level languages ! Program complexity A general approximation of speed for large datasets Counts the number of times you execute the core loop Loop twice: O(n2 ), loop thrice: O(n3 ) Remember: sorting is O(n ln(n)), list lookup O(n), btree lookup O(ln(n)), dict between O(1) and O(n), usually considered as O(ln(n)) Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Use efficient algorithm Locating CPU usage Use C code CPU optimization Do things once Memory optimization The 90-10 or 80-20 law Massive sites deployment The test case Presenting the test case We have two lists of words We want words in both lists I used Debian’s french and english wordlists Comparable uses in Plone Finding users in two groups Handling tags (for tag clouds, for example) Automatically finding related documents Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Use efficient algorithm Locating CPU usage Use C code CPU optimization Do things once Memory optimization The 90-10 or 80-20 law Massive sites deployment The test program The naive version french = open(quot;frenchquot;) english = open(quot;englishquot;) french = list(french) english = list(english) words = [ word for word in french if word in english ] print len(french), len(english), len(words) Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Use efficient algorithm Locating CPU usage Use C code CPU optimization Do things once Memory optimization The 90-10 or 80-20 law Massive sites deployment The first results Big dataset French words: 139704, English words: 98326 Program took 7m52s; replace list with set: 0.17s Speed-up factor: 2700x More reasonable dataset At 1000 words: 140ms and 4.1ms; speed-up: 34x At 100 words: 1.7ms and 0.32ms; speed-up: 5x Morality Use an efficient algorithm; correct datatype is often enough Don’t forget Python’s powerful set module ! Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Use efficient algorithm Locating CPU usage Use C code CPU optimization Do things once Memory optimization The 90-10 or 80-20 law Massive sites deployment Use C code: trust Python The theory On the other side, some people like to hand-optimize code In addition to being less readable, it’s often slower Use C code when you can; trust Python’s standard library Applying to the previous program Our lists are already sorted We can save time, and have a O(n) algorithm But the sets version is still faster: 0.17s, compared to 0.21s for my manually optimized one (that’s 23% faster) Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Use efficient algorithm Locating CPU usage Use C code CPU optimization Do things once Memory optimization The 90-10 or 80-20 law Massive sites deployment My beautiful hand-crafted code french = list(open(quot;frenchquot;)) english = list(open(quot;englishquot;)) pos = 0 maxpos = len(english) - 1 words = [] for word in french: while (word > english[pos]) and (pos < maxpos): pos += 1 if word == english[pos]: words.append(word) print len(french), len(english), len(words) Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Use efficient algorithm Locating CPU usage Use C code CPU optimization Do things once Memory optimization The 90-10 or 80-20 law Massive sites deployment Do things once The theory Avoid doing the same thing many times Use local variables to store results In a Zope context, it’s even more true with acquisition And in restricted code, with security checks, even more ! Applying to the previous program Move the maxpos from before to inside the loop We go from 0.21s to 0.23s, that’s almost 10% slower. Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Use efficient algorithm Locating CPU usage Use C code CPU optimization Do things once Memory optimization The 90-10 or 80-20 law Massive sites deployment The last golden rule The last golden rule It states that 10% of the code takes 90% of the ressources The lighter version is 20% takes 80% of the ressources It’s already used in Zope: some parts, like ZODB or security, are implemented in C But it applies to your code too ! This rule is also a nice transition. . . How to detect those ? This will be our next part ! Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Using decorators CPU optimization Issues and tips about benchmarks Memory optimization Using DeadlockDebugger Massive sites deployment The easy way: using decorators The idea Write a simple decorator printing the time taken by its function execution Use it around functions you want to test How to get more accurate values On Unix systems, you can use resource.getrusage It’ll give you user-space and kernel-space CPU usage The limits You need to decorate all your functions It’ll consider all threads, not just your own Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Using decorators CPU optimization Issues and tips about benchmarks Memory optimization Using DeadlockDebugger Massive sites deployment System, user or real ? ... I’ll say blue ! System and user time Only account for CPU time used by your Zope process System is the time spent in kernel-mode, like IO, thread handling, . . . System doesn’t account for IO waiting User is the time spent in your code, Python, Zope, Plone Real-life clock Is disturbed by other processes But accounts for time spent waiting for IO (disk spinning, ZEO server answer, swaping pages in and out, . . . ) Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Using decorators CPU optimization Issues and tips about benchmarks Memory optimization Using DeadlockDebugger Massive sites deployment Schroedinger cat will bite us ! Benchmarking affects your application You can’t observe a process (be it a cat or a Zope) without disturbing it getrlimit, time and logging uses CPU time But if you don’t abuse, the cost stays very low Benchmarking moves you away from real-life Unix doesn’t provide per-thread accounting Benchmarking a full multi-threaded system is unreliable Running a single thread is going away from real-life It can change swapping or ZEO access Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Using decorators CPU optimization Issues and tips about benchmarks Memory optimization Using DeadlockDebugger Massive sites deployment Beware the power of chaos Zope is a complex system It contains cache and lot of other stateful devices Running a bench twice can lead to different results The fresh load strategy Do all your bench on a freshly started Zope That’ll be the worse case, when all caches are empty The second test strategy Start a fresh Zope 1 Run the code once, discard the result 2 Run it again, once caches are full 3 Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Using decorators CPU optimization Issues and tips about benchmarks Memory optimization Using DeadlockDebugger Massive sites deployment The quick and dirty profiler What is DeadlockDebugger ? A useful package listing the current state of all threads Its goal was to locate deadlocks But you can use it to find where your code spent its time It is Free Software, while the pofiler module is not The magical formula ~ $ while true; do time >> bench ; lynx -dump $DEADLOCK_DEBUG_URL >> bench ; done Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Using decorators CPU optimization Issues and tips about benchmarks Memory optimization Using DeadlockDebugger Massive sites deployment Final remarks Don’t disturb the beast When you do benchmarks, try to keep your computer otherwise idle That includes between two bench ! Running Firefox or OpenOffice will screw up your memory cache Don’t mix chemicals The DeadlockDebugger trick is highly intrusive It’ll give you a good overall view, but is not very accurate Don’t mix it with serious benchmarking Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Use catalog and product code CPU optimization Using caches Memory optimization The hidden devil: conflict errors Massive sites deployment The ABC of efficient Zope code Use the catalog It prevents loading objects from the ZODB Unpickling is slow, even more when using ZEO But don’t abuse from it, either Use product code Zope security checks have a cost Page Templates and Python Scripts are slow Content types, tools, Zope3 code and external methods are fast But be careful not to create security holes ! Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Use catalog and product code CPU optimization Using caches Memory optimization The hidden devil: conflict errors Massive sites deployment A good cache will save your life How to do it ? Cache the results of (moderatly) slow methods You can write a cache decorator Cache tricks and tips Don’t store Zope objects, but simple Python types You can use the parameters as the key Don’t forget the context Consider wisely what to use as the key For example, roles or userid ? Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Use catalog and product code CPU optimization Using caches Memory optimization The hidden devil: conflict errors Massive sites deployment The evil twin of caches: expiry Automatic expiry Real-life time based Maximal number of objects LRU or scoring ? Manual expiry Easy and ugly: restarting the Zope Including a flush button somewhere Semi-automatic expiry Quite complex to handle But the most efficient Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Use catalog and product code CPU optimization Using caches Memory optimization The hidden devil: conflict errors Massive sites deployment The advertising page GenericCache A GPLed pure Python module It provides a decorator and a class Supports custom key marshallers Supports several expiry schemes (time, LRU, manual) Can easily support semi-automatic expiry Using it in Zope You need to write your own key marshallers You can integrate the flushing in your UI Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Use catalog and product code CPU optimization Using caches Memory optimization The hidden devil: conflict errors Massive sites deployment Why and how do they happen ? Threads and transactions Nothing is written to the ZODB until the end But two threads can change the same object at the same time That is a conflict: Zope will raise an error Zope default behaviour Zope will then restart the transaction The user will not see anything But the processing will be done twice... Zope will give up after three tries Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Use catalog and product code CPU optimization Using caches Memory optimization The hidden devil: conflict errors Massive sites deployment How to limit them Make changes local Try to not change too many objects in the same transaction This is a design decision, you must think about it before It is not always possible Commit your transaction You can manually commit your transaction This will lower the risk of collision But it may also break the consistency Use it on heavy processing, on safe points Example: reindexing a catalog, processing a daily batch Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Use catalog and product code CPU optimization Using caches Memory optimization The hidden devil: conflict errors Massive sites deployment How to resolve them Some conflicts may be resolved Most recent access date is the max of both Counters are easily resolved Container conflicts can sometimes be resolved too Zope allows you to do so With the _p_resolveConflict method It takes three objects: old state, save state, current state It’s up to you to do a three-way merge Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Use catalog and product code CPU optimization Using caches Memory optimization The hidden devil: conflict errors Massive sites deployment Please draw me a sheep Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Locating memory usage CPU optimization Optimizing memory Memory optimization Massive sites deployment Peeling onion’s layers Base idea Many layers can delay memory release It’s hard to know how much memory is really used Layers Python’s garbage collector may delay release of objects 1 Libraries may recycle objects (like linked-lists items) 2 Your libc’s free may delay release to the operating system 3 The operating system may delay unmap or sbrk actual 4 operation Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Locating memory usage CPU optimization Optimizing memory Memory optimization Massive sites deployment A bit of theory about swap Swap and memory In modern operating systems, the memory is just a cache for disk Swapping, unmapping binaries or releasing cache is the same You can have swap used without it being a problem Working set and trashing The problem is in frequency, not amount During an operation, a set of pages is used, called the “working set” When it doesn’t fit in memory, your system is “trashing” Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Locating memory usage CPU optimization Optimizing memory Memory optimization Massive sites deployment So, do I have a memory problem ? The vmstat command Gives regular information about virtual memory system Exists on most Unixes, including GNU/Linux and FreeBSD Typical invocation is vmstat 2 Most interesting columns are si and so on Linux, pi and po on FreeBSD More informations It is a global command, accounting for the whole system It also contains “normal” I/O to discs and CPU state Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Locating memory usage CPU optimization Optimizing memory Memory optimization Massive sites deployment A few tips Using the gc module You can check for uncollectable cycles You can ask for manual collection You can get a list of all objects Warning, basic types like string are not returned, you need to walk in containers Monkey-patching malloc You can track malloc/free calls with ltrace On the GNU C library, you can write hooks to malloc in C Tools like dmalloc can provide debug logs Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Locating memory usage CPU optimization Optimizing memory Memory optimization Massive sites deployment Sometimes, you get both at once Use the catalog Using the catalog instead of awakening objects will reduce memory usage It’ll also reduce disk usage, and therefore the cost of swapping Store big files in the filesystem It’ll avoid polluting ZODB cache Even better: use Apache to serve them directly You can use products like tramline You can just serve with Apache the BlobFile directory Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Locating memory usage CPU optimization Optimizing memory Memory optimization Massive sites deployment Reference counting and big memory chunks The simple theory Objects are only destroyed when there is no reference Local variables are destroyed at the end of the function You can use the del operator to force it Use it on big objects before any slow operation But sometimes it’s not that simple A reference can held in a calling function You cannot destroy it then You need to pass by reference Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Locating memory usage CPU optimization Optimizing memory Memory optimization Massive sites deployment Let’s find a perfect culprit The code def process(target): what = generate_data() send(target, what) def send(target, what): encoded = encode(what) del what target.write(encoded) Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage Locating memory usage CPU optimization Optimizing memory Memory optimization Massive sites deployment A solution The code def process(target): what = generate_data() params = { quot;dataquot;: what } del what send(target, params) def send(target, params): encoded = encode(params[quot;dataquot;]) del params[quot;dataquot;] target.write(encoded) Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage ZEO pro and cons CPU optimization Squid caching overview Memory optimization Site exporting strategies Massive sites deployment The two faces of ZEO The good face of ZEO It is very easy to set up It makes Zope parellizable on many CPUs It supports networking, and can allow clustering It can do both failover and load-balancing The bad face of ZEO It’ll slow any ZODB access The database itself is centralised and can be a bottleneck Session objects will need to go through ZODB marshalling too Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage ZEO pro and cons CPU optimization Squid caching overview Memory optimization Site exporting strategies Massive sites deployment Using a proxy-cache Do things only once The rendering a Plone page is complex But many visitors will have exactly the same results Using a proxy-cache like Squid (or Apache) can avoid doing it many times But that only covers anonymous users Squid can do varying You can use multilingual site behind a Squid It’ll detect the headers are the serve the correct version from the cache Example: EADS Astrium Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage ZEO pro and cons CPU optimization Squid caching overview Memory optimization Site exporting strategies Massive sites deployment If one ZODB is not enough... Static HTML export Not too hard to implement, but still tricky Removes all dynamic features like search Exporting a site You can copy a ZODB from a running Zope Or you can use zexp and XML-RPC or Post Resyncing back It requires some hacks But we did it on forum (PloneGossip) and event subscription (SignableEvent) Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app
  • The golden rules Locating CPU usage ZEO pro and cons CPU optimization Squid caching overview Memory optimization Site exporting strategies Massive sites deployment Conclusion Plone is not doomed to be slow ! Optimization has to be part of all steps, from design to code Even with fast CPUs, it’s still an important issue Thanks To Pilot Systems team for feedback To the Free Software community for all the tools available The usual... Any question ? Gaël LE MIGNOT – Pilot Systems How to minimize CPU and memory usage of Zope and Plone app