Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PyCon US 2013 Making Apache suck less for hosting Python web applications


Published on

It is not hard to find developers who will tell you that Apache sucks for running Python web applications. Is there a valid basis to such claims or have they simply been misguided by the views of others? This talk will endeavor to shine a light on the realities of and limitations in working with Apache, as well as the challenges in implementing the mod_wsgi module for Apache.

  • Be the first to comment

PyCon US 2013 Making Apache suck less for hosting Python web applications

  1. 1. Making Apache suck less for hosting Python web applications Graham Dumpleton PyCon US - March 2013Saturday, 16 March 2013
  2. 2. Following Along Slides contains presenter notes!Saturday, 16 March 2013If you want to follow along with the slides for this talk onyour laptop, you can view them on Theslides do have my presenter notes, so you can also readit all again later if you are too busy surfing the net, ordidnt catch some point.
  3. 3. Apache Sucks • Is too hard to configure. • Is bloated and uses heaps of memory. • Is slow and doesnt perform very well. • Is not able to handle a high number of concurrent requests.Saturday, 16 March 2013These days there seems to be a never ending line ofpeople who will tell you that Apache sucks and that youare crazy if you use it, especially for hosting Python webapplications. What is the truth? Do the people who makesuch claims actually understand how Apache works orare they just repeating what someone else told them?
  4. 4. Apache Sucks • Is too hard to configure. • Is bloated and uses heaps of memory. • Is slow and doesnt perform very well. • Is not able to handle a high number of concurrent requests.Saturday, 16 March 2013As the author of the mod_wsgi module for Apache, whatI want to do in this talk is go through and look at whatsome of the pain points are when configuring Apache torun Python web applications. The intent is that you canwalk away with a bit more insight into how Apache worksand what is required to properly setup Apache andmod_wsgi. So, if you like, I am going to explain how tomake Apache suck less.
  5. 5. Where Has All My Memory Gone • Reasons for excessive memory usage. • Python web applications are fat to start with. • Poor choice of multiprocessing module (MPM). • Poor choice of configuration for the MPM used. • Loading of Apache modules you arent using. • Size of Apache memory pools for each thread. • Inability to benefit from copy on write.Saturday, 16 March 2013The biggest criticism which seems to be levelled atApache is that it is bloated and uses too much memory.There are various reasons Apache can use a lot ofmemory. Many of these are under the control of the userand not necessarily a failing of Apache though.
  6. 6. Why Is Response Time So Bad • Reasons for slow response times. • Not enough capacity configured to handle throughput. • Keep alive causing artificial reduction in capacity. • Machine slowing down due to frequent process recycling. • High cost of loading WSGI applications on process startup.Saturday, 16 March 2013Another criticism is why is Apache so slow. Like withmemory usage, this can also have a lot to do with howApache has been configured. In practice, if Apache issimply setup properly for the specifics of runningdynamic Python web applications, and takes intoconsideration the constraints of the system it is beingrun on, neither of these should be an issue.
  7. 7. Streamlining The Apache Installation LoadModule authz_host_module modules/ LoadModule mime_module modules/ LoadModule rewrite_module modules/ LoadModule wsgi_module modules/mod_wsgi.soSaturday, 16 March 2013The first thing one can do is to strip down what modulesApache is loading. Because Apache is a workhorse thatcan be used for many different tasks, it comes with arange of pluggable modules. There is likely going to beany number of modules getting loaded you arent using.To cut down on base memory used by Apache itself, youshould disable all Apache modules you are not using.
  8. 8. Python Web Applications Are Fat • Raw Web Server • Apache - Streamlined (2 MB) • Python WSGI Hello World • Apache/mod_wsgi - Streamlined (5 MB) • Apache/mod_wsgi - Kitchen Sink (10MB) • Gunicorn - Sync Worker (10MB) • Real Python Web Application • Django (20-100MB+)Saturday, 16 March 2013Beyond the server, it has to be recognised that any useof Python will cause an immediate increase in memoryused. Load a typical web application, along with all themodules from the standard library it requires, as well asthird party modules and memory use will grow quitequickly. The actual base memory consumed by the webserver at that point can be quite small in comparison.
  9. 9. Python Web Applications Are Fat • Raw Web Server • Apache - Streamlined (2 MB) • Python WSGI Hello World • Apache/mod_wsgi - Streamlined (5 MB) • Apache/mod_wsgi - Kitchen Sink (10MB) • Gunicorn - Sync Worker (10MB) • Real Python Web Application • Django (20-100MB+)Saturday, 16 March 2013Overall, it shouldnt really matter what WSGI server youuse, the Python interpreter and the Python webapplication itself should always use a comparableamount of memory for a comparable configuration. Thelaws of nature dont suddenly change when you startusing Apache to host a Python web application. Memoryused by the Python web application itself in a singleprocess should not suddenly balloon out for no reason.
  10. 10. Processes Vs Threads Server Browser Parent Client Server Server Server Processes Worker Worker Worker ThreadsSaturday, 16 March 2013An appearance therefore of increased memory usage ismore likely going to be due to differences in the serverarchitecture. When I say server architecture, I specificallymean the mix of processes vs threads that are used bythe server hosting the WSGI application to handlerequests. Using processes in preference to threads willobviously mean that more memory is being used.
  11. 11. Different Server Architectures • Apache • Prefork MPM - Multiple single threaded processes. • Worker MPM - Multiple multi threaded processes. • WinNT MPM - Single multi threaded processes. • Gunicorn • Sync Worker - Multiple single threaded processes. MPM = Multiprocessing ModuleSaturday, 16 March 2013The big problem in this respect is that beginners knowno better and will use whatever the default configurationis that their server distribution provides. For Apache,which is often supplied with the prefork multiprocessingmodule, or MPM, this can very easily cause problems,because it uses single threaded processes.
  12. 12. Default Server Configuration • Apache • Prefork MPM - 150 processes (maximum) / 1 thread per process. • Worker MPM - 6 processes (maximum) / 25 threads per process. • WinNT MPM - 1 process (fixed) / 150 threads per process. • Gunicorn • Sync Worker - 1 process (fixed) / 1 thread per process.Saturday, 16 March 2013Although prefork MPM may only initially start out with asingle process, it can automatically scale out to 150processes. That is 150 copies of your Python webapplication. At 20MB per process that is already 3GB andat 20MB that would be considered a small Python webapplication. In contrast, gunicorn sync worker defaults toa single process and single thread and doesnt scale. Thememory requirement of gunicorn would therefore staythe same over time.
  13. 13. Making Apache Suck Less (1) • Dont use Apache default configurations for hosting Python web applications. • Dont use the prefork MPM unless you know how to configure Apache properly, use the worker MPM, it is more forgiving. • Dont allow Apache to automatically scale out the number of processes over too great a range. • Dont try and use a single Apache instance to host Python, PHP, Perl web applications at the same time.Saturday, 16 March 2013So, whatever you do, dont use the default configurationthat comes with your server distribution. For Python webapplications you generally cant avoid having to tune it.This is because the Apache defaults are setup for staticfile serving and PHP applications. Especially dont try anduse the same Apache instance to host Python webapplications at the same time as running PHP or Perlapplications as each has different configurationrequirements.
  14. 14. What MPM Are You Using? $ /usr/sbin/httpd -V | grep Server MPM Server MPM: PreforkSaturday, 16 March 2013How do you work out which MPM you are using? Prior toApache 2.4 the type of MPM being used was definedwhen Apache was being compiled and was staticallylinked into the Apache executable. From Apache 2.4, theMPM can also be dynamically loaded and so defined atruntime by the Apache configuration. Either way, you candetermine the MPM in use by running the Apacheexecutable with the -V option.
  15. 15. WSGI multiprocess/multithread wsgi.run_once wsgi.multiprocess wsgi.multithread CGI TRUE TRUE FALSE Prefork FALSE TRUE FALSE Worker FALSE TRUE TRUE WinNT FALSE FALSE TRUESaturday, 16 March 2013Another way of determining the specific processarchitecture in use is by consulting the multiprocess andmultithread attributes passed in the WSGI environ witheach request. Neither of these though will actually tellyou how many processes or threads are in use. For thatyou need to start looking at the Apache configurationitself.
  16. 16. Defaults From Configuration File extra/httpd-mpm.conf <IfModule mpm_prefork_module> <IfModule mpm_worker_module> StartServers 1 StartServers 2 MinSpareServers 1 MaxClients 150 MaxSpareServers 10 MinSpareThreads 25 MaxClients 150 MaxSpareThreads 75 MaxRequestsPerChild 0 ThreadsPerChild 25 MaxRequestsPerChild 0 </IfModule> </IfModule>Saturday, 16 March 2013This is where we can actually end up in a trap. For thestandard Apache configuration as provided with theApache Software Foundations distribution, althoughthere are example MPM settings provided, thatconfiguration file isnt actually included by default. Thesettings in that file are also different to what is compiledinto Apache, so you cant even use it as a guide to whatthe compiled in defaults are.
  17. 17. Compiled In Prefork MPM Settings StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 256 MaxRequestsPerChild 10000Saturday, 16 March 2013So although I said before that the prefork MPM couldscale up to 150 processes automatically, that was on theassumption that the default settings in the Apacheconfiguration file were actually used. If those settingsarent used, then it is instead 256 processes, making iteven worse. Some people recommend throwing away thedefault configuration files and starting from scratchmeaning one uses the more lethal compiled in settings.
  18. 18. Meaning Of Prefork Settings • StartServers - Number of child server processes created at startup. • MaxClients - Maximum number of connections that will be processed simultaneously. • MaxRequestsPerChild - Limit on the number of requests that an individual child server will handle during its life.Saturday, 16 March 2013For those who are not familiar with these settings, whatdo they actually mean. StartServers is the initial numberof processes created to handle requests. Because preforkuses single threaded processes, the maximum numberof processes ends up being dictated by MaxClients.
  19. 19. Automatic Scaling In Prefork • MinSpareServers - Minimum number of idle child server processes. • MaxSpareServers - Maximum number of idle child server processes.Saturday, 16 March 2013The settings which need more explanation are the minand max spare processes. The purpose of these is tocontrol how Apache dynamically adjusts the number ofprocesses being used to handle requests.
  20. 20. Algorithm For Scaling In Prefork if idle_process_count > max_spare_servers: kill a single process elif idle_process_count < min_spare_servers: spawn one or more processesSaturday, 16 March 2013In very simple terms, what Apache does is wake up eachsecond and looks at how many idle processes it has atthat point which are not handling requests. If it has moreidle processes than the maximum specified, it will kill offa single process. If it has less idle processes than theminimum spare required it will spawn more. How many itspawns will depend on whether it had spawned any inthe previous check and whether it is creating themquickly enough.
  21. 21. Visualising Scaling In Prefork Concurrent Requests Process Creation Total No Of Processes Idle ProcessesSaturday, 16 March 2013When presented with the MPM settings being used, evenI have to think hard sometimes about what the result ofthose settings will be. To make it easier to understand, Irun the settings through a simulator and chart theresults. In this example the number of concurrentrequests is ramped up from zero and when maximumcapacity is reached, it is ramped back down to zeroagain.
  22. 22. Visualising Scaling In Prefork Concurrent Requests Create Kill Process Creation Total No Of Processes Idle ProcessesSaturday, 16 March 2013By using a simulator and visualising the results, itbecomes much easier to understand how Apache willbehave. We can see when Apache would create processesor kill them off. What the total number of processes willbe as it scales, and how that is being driven by trying toalways maintain a pool of idle processes within thebounds specified.
  23. 23. Floor On The Number Of Processes MaxSpareServers MinSpareServersSaturday, 16 March 2013Zooming in on the chart for the total number ofprocesses available to handle requests, one thing thatstands out for example is that there is an effective flooron the number of processes which will be kept around.No processes will be killed if we are at or below this level.
  24. 24. Starting Less Than The Minimum StartServers 1 MinSpareServers 5 MaxSpareServers 10 MaxClients 256Saturday, 16 March 2013Too often, people who have no idea how to configureApache get in and start mucking around with thesevalues, not understanding the implications of what theyare doing. The simulator is great in being able to give aquick visual indicator as to whether something is amiss.One example is where the number of servers to bestarted is less than the minimum number of spareservers.
  25. 25. Delayed Started Of Processes Creating ProcessesSaturday, 16 March 2013What happens in this case is that as soon as Apachestarts doing its checks, it sees that it isnt actuallyrunning enough processes to satisfy the requirement forthe minimum number of idle processes. It thereforestarts creating more, doubling the number each timeuntil it has started enough. Rather than actually startingall processes immediately, in this case it takes 3 secondsthus potentially limiting the initial capacity of the server.
  26. 26. Starting Up Too Many Servers StartServers 100 MinSpareServers 5 MaxSpareServers 10 MaxClients 256Saturday, 16 March 2013At the other end of the scale, we have people whochange the number of servers to be started to be greaterthan the maximum spare allowed.
  27. 27. Immediate Kill Off Of Processes Killing ProcessesSaturday, 16 March 2013This time when Apache starts doing its checks, if finds ithas more idle processes than allowed and starts killingthem off at a rate of 1 per second. Presuming no trafficcame in that necessitated those processes actuallyexisting, it would take over a minute to kill off all theexcess processes.
  28. 28. Making Apache Suck Less (2) • Ensure MaxSpareServers is greater than MinSpareServers. If you dont, Apache will set MaxSpareServers to be MinSpareServers+1 for you anyway. • Dont set StartServers to be less than MinSpareServers as it will delay start up of processes so as to reach minimum spare required. • Dont set StartServers to be greater than MaxSpareServers as processes will start to be killed off immediately.Saturday, 16 March 2013To avoid such delayed process creation, or immediatekilling off of processes on startup, you should ensurethat the value of StartServers is bounded byMinSpareServers and MaxSpareServers.
  29. 29. Overheads Of Process Creation • Initialisation of the Python interpreter. • Loading of the WSGI application. • Loading of required standard library modules. • Loading of required third party modules. • Initialisation of the WSGI application.Saturday, 16 March 2013Why do we care about unnecessary process creation?After all, arent the processes just a fork of the Apacheparent process and so cheap to create? The problem isthat unlike mod_php where PHP is initialised and allextension modules preloaded into the Apache parentprocess, when using mod_wsgi, Python initialisation isdeferred until after the processes are forked. Anyapplication code and required Python modules are thenlazily loaded.
  30. 30. Preloading Is A Security Risk • You dont necessarily know which WSGI application to load until the first request arrives for it. • Python web applications arent usually designed properly for preloading prior to forking of worker processes. • All code run in the Apache parent process is run as root.Saturday, 16 March 2013If initialising Python and loading the WSGI application inthe worker process can be expensive, why cant wepreload everything in the Apache parent process beforethe worker processes are forked? Even if a Python webapplication were designed to be able to be preloadedand run properly after the process was forked, the keyissue is that users application code on startup would runas root if executed in the parent process and that is onevery big security risk.
  31. 31. Preloading Causes Memory Leaks • The Python interpreter will leak memory into the parent process when an Apache restart occurs.Saturday, 16 March 2013Add to that, because of what Python does (or should Isay doesnt do) when the interpreter is destroyed,combined with the way in which Apache reloadsmod_wsgi when restarting, the Python interpreter willleak memory into the Apache parent process. If Apacherestarts are done on a regular basis, the size of theApache parent will keep growing over time and thus sowill the forked worker processes as well.
  32. 32. Downsides Of Not Preloading • Additional CPU load when creating new worker process. • Worker processes will not be immediately ready. • No saving in memory usage from copy on write.Saturday, 16 March 2013So running within Apache we have no choice and have todefer initialisation of Python and loading of the WSGIapplication until after the child processes are forked.This causes additional CPU load each time a process isstarted up and the time taken will also mean thatrequests will be held up. Finally, because we are notpreloading in the parent, we cannot benefit from reducedmemory usage from copy on write features of theoperating system.
  33. 33. Avoiding Process CreationSaturday, 16 March 2013Process startup is therefore expensive. We want to avoiddoing it and certainly dont want it occurring at a timewhich is inconvenient. Unfortunately, especially withprefork MPM, allowing Apache to dynamically createprocesses can result in a lot of process churn if you arenot careful. The more problematic situation is wherethere is a sudden burst off traffic and there is notenough processes already running to handle it.
  34. 34. Sudden Burst In Traffic Sudden need to create processes.Saturday, 16 March 2013In the worst case scenario, the increased load fromcreating processes when a sustained traffic spike occurs,could see the whole system slow down. The slow downcan make Apache think it isnt creating enoughprocesses quickly enough, so it keeps creating more andmore. Pretty quickly it has created the maximum numberof processes, with the combined CPU load of loading theWSGI application for all of them, causing the server togrind to a halt.
  35. 35. Constant Churn Of Processes Continual killing off and creation of processes.Saturday, 16 March 2013Even after the processes have been created we can stillsee process churn. This is because Apache doesnt lookat request load over time, it only looks at the number ofconcurrent requests running at the time of the check.This can actually bounce around quite a lot each second.There will therefore be a continual churn of processes asApache thinks there is more than required and kills someoff and then when it again believes it does not haveenough and creates more.
  36. 36. Raising The Floor StartServers 5 MinSpareServers 5 MaxSpareServers 150 MaxClients 256Saturday, 16 March 2013What if we raise that floor on the number of processes asdetermined by the MaxSpareServers setting? We said thatso long as the number of processes was below this level,none would be killed off. Lets try then setting that to alevel above the average number of processes in use.
  37. 37. Number Of Processes Plateaus Initial Spike Reduced Incidence Of Process CreationSaturday, 16 March 2013What will happen is that although there will be an initialspike in the number of processes created, after thatthere will only be a slow increase as the number ofprocesses finds its natural level and plateaus. So long aswe stay below MaxSpareServers we will avoid processchurn.
  38. 38. Breaching The Maximum Falls Back To Maximum After SpikeSaturday, 16 March 2013We havent set the maximum number of spare processesto the maximum allowed clients quite yet though. So it isstill possible that when a spike occurs we will createmore than the maximum spare allowed. When trafficrecedes, the number of processes will reduce back to thelevel of the maximum and no further. Finding theoptimal level for the maximum spare processes to avoidchurn can be tricky, but one can work it out bymonitoring utilisation.
  39. 39. Maximum Number Of Requests MaxRequestsPerChild 0Saturday, 16 March 2013Do be aware though that all this work in tuning thesettings can be undone by the MaxRequestsPerChildsetting. We want to avoid process churn. It is no goodsetting this to such a low value that this would causeprocess recycling after a very short period of time, as itjust reintroduces process churn in another way. It isbetter to have this be zero resulting in processes stayingpersistent in memory until shutdown.
  40. 40. Handling Large Number Of Clients MaxClients 256Saturday, 16 March 2013Now the only reason that the defaults for Apache specifysuch a large value for MaxClients, and thus a largenumber of processes when single threading is used, isbecause of slow clients and keep alive. A high number isrequired to support concurrent sessions from manyusers. If using single threaded processes though, thismeans you will need to have much more memoryavailable.
  41. 41. Front End Proxy And KeepAlive nginx Server Browser Front Parent Client End Server Server Server Processes Worker Worker WorkerSaturday, 16 March 2013A much better solution is to put a nginx proxy in front ofApache. The nginx server will isolate Apache from slowclients as it will only forward a request when it iscompletely available and can be handled immediately.The nginx server can also handle keep alive connectionsmeaning it can be turned off in Apache. This allows us tosignificantly reduce the number of processes needed forApache to handle the same amount of traffic as before.
  42. 42. Making Apache Suck Less (3) • Set MaxSpareProcesses at a level above the typical number of concurrent requests you would need to handle. • Do not use MaxRequestsPerChild, especially at a low count which would cause frequent process churn. • Remember that if you dont set MaxRequestsPerChild explicitly, it defaults to 10000. • Use nginx as a front end proxy to isolate Apache from slow clients. • Turn off keep alive in Apache when using nginx as a front end.Saturday, 16 March 2013Key in eliminating unwanted CPU usage was thereforeavoiding process churn which is achieved by adjustingthe maximum allowed number of spare processes andensuring we arent periodically recycling processes for nogood reason. We can though also reduce the number ofprocesses we need in the first place by adding nginx as aproxy in front of Apache.
  43. 43. Prefork MPM Vs Worker MPM Prefork MPMSaturday, 16 March 2013All of what I have explained so far focused on preforkMPM. I have concentrated on it because it magnifies theproblems that can arise. When people say Apache sucksit is usually because they were using prefork MPM withan inadequate configuration. Use of prefork MPM with anginx proxy will give you the best performance possibleif setup correctly. As I said before though, using workerMPM is much more forgiving of you having a poor setup.
  44. 44. Visualising Scaling In Prefork Worker MPMSaturday, 16 March 2013The reasons for this are that for the same large defaultvalue of MaxClients, worker MPM will use a lot lessprocesses than prefork MPM. This is because eachprocess will have 25 threads handling requests insteadof 1. Being less processes, worker MPM will therefore seeless copies of your Python web application and so lessmemory usage.
  45. 45. Compiled In Worker MPM Settings StartServers 3 MinSpareThreads 75 MaxSpareThreads 250 ThreadsPerChild 25 MaxClients 400 MaxRequestsPerChild 10000Saturday, 16 March 2013In the case of worker MPM, by default 3 processes wouldbe started initially with the compiled in defaults. WithMaxClients of 400 and ThreadsPerChild being 25, thatmeans a maximum of 16 processes would be created.
  46. 46. Automatic Scaling In Worker • MinSpareThreads - Minimum number of idle threads available to handle request spikes. • MaxSpareThreads - Maximum number of idle threads.Saturday, 16 March 2013Settings related to scaling when using worker MPM referto threads whereas with prefork MPM they were in termsof processes. MaxSpareThreads defaults to 250, whichequates to the equivalent of 10 processes.
  47. 47. Making Apache Suck Less (4) • Ensure MaxSpareThreads is at at least MinSpareThreads +ThreadsPerChild. If you dont, Apache will set it to that for you anyway. • Suggested that MinSpareThreads and MaxSpareThreads be set as multiples of ThreadsPerChild. • Dont set StartServers to be less than MinSpareThreads/ ThreadsPerChild as it will delay start up of processes so as to reach minimum spare required. • Dont set StartServers to be greater than MaxSpareThreads/ThreadsPerChild as processes will start to be killed off immediately.Saturday, 16 March 2013One very important thing to note, is that although theseare expressed in terms of threads, Apache doesnt scaleat the thread level. The number of threads per process isstatic. When scaling it is the same as prefork, a processwill either be created or killed. The decision though isbased on available threads instead.
  48. 48. Worker Defaults More Forgiving Initial Spike Is For Much Fewer Processes Followed By No Churn At AllSaturday, 16 March 2013Running our simulation of random traffic from beforewith a similar level of concurrent requests and althoughwe still had a initial spike in creating processes, no newprocesses were needed after that, as we were within thelevel specified by max spare threads. No churn means nowasted CPU through continually creating and killingprocesses. Using the compiled in defaults at least, this iswhy worker MPM is more forgiving that prefork MPM.
  49. 49. Reducing Per Thread Memory Use • MaxMemFree - Maximum amount of memory that the main allocator is allowed to hold without calling free(). MaxMemFree 256 # KBytesSaturday, 16 March 2013As before, and especially if using nginx as a front endproxy, one can adjust MaxClients, min and max sparethreads and perhaps bring down even further theamount of resources used. A more important settingthough is MaxMemFree. This is the maximum amount ofmemory the Apache per thread memory pool is allowedto hold before calling free on memory. Prior to Apache2.4, this was unbounded. In Apache 2.4 it is 2MB.
  50. 50. Making Apache Suck Less (5) • Ensure that MaxMemFree is set and not left to be unbounded. • Even on Apache 2.4 where is 2MB, consider reducing the value further.Saturday, 16 March 2013Even at 2MB in Apache 2.4, this could mean that for 25threads, 50MB can be held by the persistent memorypools in each process. When running mod_wsgi, undernormal circumstances, there should not be much call formemory to be allocated from the per request memorypool. To be safe though you should ensure MaxMemFreeis set and with a reduced value if possible.
  51. 51. Daemon Mode Of Apache/mod_wsgi Server Browser Parent Client Server Server Server Processes Worker Worker Worker Daemon Process(es) Daemon Process ThreadsSaturday, 16 March 2013Now the configuration for prefork or worker MPM areprincipally an issue when using what is called embeddedmode of mod_wsgi. That is, your WSGI application runsinside of the Apache server child worker processes. Thedynamic scaling algorithm of Apache being what cancause us grief when doing this. Using worker MPM helps,but an even safer alternative is to use mod_wsgi daemonmode instead. In this case your WSGI application runs ina separate set of managed processes.
  52. 52. Daemon Mode Configuration WSGIDaemonProcess myapp processes=3 threads=5 WSGIScriptAlias / /some/path/ process-group=myapp application-group=%{GLOBAL}Saturday, 16 March 2013The main difference when using daemon mode is thatthere is no automatic scaling of the number ofprocesses. The number of processes and threads isinstead fixed. Being fixed everything is more predictableand you only need to ensure you have sufficient capacity.Using daemon mode, the need to have nginx as a frontend is reduced as the Apache server child workerprocesses are serving much the same process inisolating the WSGI application from slow clients.
  53. 53. Exclusively Using Daemon Mode • WSGIRestrictEmbedded - Controls whether the Python interpreter is initialised in Apache server worker processes. WSGIRestrictEmbedded OnSaturday, 16 March 2013Because the Apache server processes are now only actingas a proxy, forwarding requests to the mod_wsgidaemon process, as well as serving static files, we dontneed to initialise the Python interpreter in the Apacheserver processes. Process creation is again lightweightand we have side stepped the need to pay so muchattention to the Apache MPM settings.
  54. 54. The Things That Make Apache Suck • An algorithm for dynamically scaling processes which isnt particularly suited to embedded Python web applications. • Default MPM and settings which magnify the issues which can arise with dynamic scaling when running Python web applications. • A concurrency mechanism that can use a lot of memory for a high number of concurrent requests, especially around handling of keep alive connections. • Defaults for memory pool sizes which cause Apache to be heavyweight on memory usage.Saturday, 16 March 2013So Apache can certainly be a challenging environment forrunning Python web applications. The main pain pointsare how its algorithm for dynamic scaling works andmemory requirements to support high concurrency. Withcareful attention it is possible though to configureApache to reduce the problems these can cause.
  55. 55. Application Performance MonitoringSaturday, 16 March 2013The simulator I demonstrated can be used to try andvalidate any configuration before you use it, but therandom nature of web site traffic means that it will notbe conclusive. This is where live monitoring of traffic inyour production web site provides a much better level offeedback. New Relic is obviously the package I would liketo see you using, but any monitoring is better than none.
  56. 56. Capacity AnalysisSaturday, 16 March 2013In New Relic, one of the reports it generates which isparticularly relevant to coming up with the bestprocesses/threads configuration is its capacity analysisreport. From this report one can see whether you haveprovided enough capacity, or whether you have overallocated and so wasting memory, or are running yourapplication over more hosts than you need and thereforepaying more money for hosting than you need.
  57. 57. Capacity AnalysisSaturday, 16 March 2013Although this talk has been about Apache/mod_wsgi,this report is just as relevant to other WSGI hostingmechanisms, such as gunicorn and uWSGI. Working atNew Relic and being able to see data coming in from awide variety of deployments it is really quite amazinghow poorly some servers are being set up, and not justwhen Apache is being used. So if you are using NewRelic, I would really suggest paying a bit more attentionto this report. Doing so can help you make your serverrun better and possibly save you money as well.
  58. 58. More Information • Slides (with presenter notes). • • Apache/mod_wsgi mailing list (preferred contact point). • • New Relic (Application Performance Monitoring) • • (special 30 day promo code - pycon13) • Personal blog posts on Apache/mod_wsgi and WSGI. • • If you really really must bother me directly. • • @GrahamDumpletonSaturday, 16 March 2013And that is all I want to cover today. If you are after moreinformation, especially if you are interested in thesimulator I demonstrated, keep an eye on my blog formore details of that sometime in the near future. If youare interested in using New Relic to better configure yourWSGI server, then you can catch me in the expo hall afterthe talk. Questions?