Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Web Server                                      Bottlenecks and                                    Performance Tuning     ...
Follow along✴http://www.slideshare.net/GrahamDumpleton
The big picture
Front end
Web applicationWeb application   Front end time 0.15 seconds      3.1 seconds
Performance Golden Rule    "80-90% of the end-user response time is spent on the     frontend. Start there."http://www.ste...
Application breakdown
Are benchmarks stupid?http://nichol.as/benchmark-of-python-web-servers
Benchmarks as a tool✴Web server benchmarks are of more value when used as an exploratory tool to understand how a specific ...
What about load tests?✴Hitting a site with extreme load will only show you that it will likely fail under a denial of serv...
What should you test?✴You should use a range of purpose built tests to trigger certain scenarios.✴Use the tests to explore...
Environment factors✴Amount of memory available.✴Number of processors available.✴Use of threads vs processes.✴Python global...
Client impacts✴Slow HTTP browsers/clients.✴Browser keep alive connections.
Application requirements✴Need to handle static assets.
Use cases to explore✴Memory used by web application.✴Using processes versus threads.✴Impacts of long running requests.✴Res...
Memory usage                  1 process     1 process                 1000 threads   1 threadhttp://nichol.as/benchmark-of...
What effects memory use?✴Web server base memory usage.✴Web server per thread memory usage.✴Application base memory usage. ...
Processes Vs Threads            150            135            120            105             90             75Processes   ...
Apache/mod_wsgi defaults   Configuration       Max Processes   Threads  Apache (prefork)mod_wsgi (embedded)       150      ...
Other WSGI serversConfiguration     Max Processes   Threads  FASTCGIflup (prefork)         50           1  FASTCGIflup (threa...
Less than fair            150            135                              Apache (prefork)            120             mod_...
What to use?✴Number of overall threads dictated by: ✴Number of concurrent users. ✴Response time for requests.✴Processes pr...
Thread utilisation                    1 second          6          5Threads          4          3          2          1
Request backlog                      Backlog occurred and queue                       time increased to 750 ms150ms       ...
Processes are better                             Backlog only started                       at higher throughput and queue...
CPU bound  Bulk of time is from doingthings within the process itself
I/O wait Waiting on responses frombackend services a significant    proportion of time
Long running requests✴Complex calculations.✴Slow backend services.✴Large file uploads.✴Large responses.✴Slow HTTP clients.
Varying request timesAverage:   1385 msMinimum:   4.7 msMaximum:   20184 msStd Dev:   3896 ms
Performance breakdownWhy is creating the connectionto PostgreSQL taking up 40%   of overall response time
Slow HTTP clients✴Add nginx as a front end to the WSGI server.✴Brings the following benefits to the WSGI server. ✴Isolation...
Request funnelling  nginxfront endApacheworkersmod_wsgidaemons
Complete overload
Forced restarts✴Triggers for restarts: ✴Manual restart to fix issues/configuration. ✴Maximum number of requests reached. ✴Re...
Auto scaling✴Apache/mod_wsgi embedded mode. ✴Apache prefork MPM defaults.  ✴Initial 1 / Maximum 150 ✴Apache worker MPM def...
Pre load everything✴Start maximum processes up front.✴Pre load your web application when the process starts and not lazily...
Horizontal scaling✴Using more servers is fine.✴Load balance across dedicated hosts.✴Or add additional hosts as required.✴En...
Monitoring is key✴Treat your server as a black box and you will never know what is going on inside.
Server monitoring✴Open source tools. ✴Monit ✴Munin ✴Cacti ✴Nagios
Python web tools✴Django debug toolbar. ✴Only useful for debugging a single request  in a development setting.✴Sentry. ✴Use...
New Relic APM
Apache/mod_wsgi
Summing up✴Use benchmarks to explore a specific system, not to compare different systems.✴Dont trust the defaults of any se...
Try New Relic✴ Graham.Dumpleton@gmail.com✴ http://www.slideshare.net/GrahamDumpleton✴ Find out more about New Relic:  ✴ ht...
Upcoming SlideShare
Loading in …5
×

PyCon US 2012 - Web Server Bottlenecks and Performance Tuning

11,689 views

Published on

Published in: Technology
  • Be the first to comment

PyCon US 2012 - Web Server Bottlenecks and Performance Tuning

  1. 1. Web Server Bottlenecks and Performance Tuning Graham Dumpleton PyCon – March 2012Graham Dumpleton @GrahamDumpletonStarting my PyCon talk. Lets hope I dont loose myvoice completely while doing this.
  2. 2. Follow along✴http://www.slideshare.net/GrahamDumpleton
  3. 3. The big picture
  4. 4. Front end
  5. 5. Web applicationWeb application Front end time 0.15 seconds 3.1 seconds
  6. 6. Performance Golden Rule "80-90% of the end-user response time is spent on the frontend. Start there."http://www.stevesouders.com/blog/2012/02/10/ the-performance-golden-rule/
  7. 7. Application breakdown
  8. 8. Are benchmarks stupid?http://nichol.as/benchmark-of-python-web-servers
  9. 9. Benchmarks as a tool✴Web server benchmarks are of more value when used as an exploratory tool to understand how a specific system works, not to compare systems.
  10. 10. What about load tests?✴Hitting a site with extreme load will only show you that it will likely fail under a denial of service attack.✴Your typical web server load test isnt alone going to help you understand how a web server is contributing to it failing.
  11. 11. What should you test?✴You should use a range of purpose built tests to trigger certain scenarios.✴Use the tests to explore corner cases and not just the typical use case.
  12. 12. Environment factors✴Amount of memory available.✴Number of processors available.✴Use of threads vs processes.✴Python global interpreter lock (GIL)
  13. 13. Client impacts✴Slow HTTP browsers/clients.✴Browser keep alive connections.
  14. 14. Application requirements✴Need to handle static assets.
  15. 15. Use cases to explore✴Memory used by web application.✴Using processes versus threads.✴Impacts of long running requests.✴Restarting of server processes.✴Startup costs and lazy loading.
  16. 16. Memory usage 1 process 1 process 1000 threads 1 threadhttp://nichol.as/benchmark-of-python-web-servers
  17. 17. What effects memory use?✴Web server base memory usage.✴Web server per thread memory usage.✴Application base memory usage. ✴Is application loaded prior to forking?✴Per request transient memory usage.
  18. 18. Processes Vs Threads 150 135 120 105 90 75Processes 60 45 30 15 0 0 15 30 45 60 75 90 105 120 135 150 Threads
  19. 19. Apache/mod_wsgi defaults Configuration Max Processes Threads Apache (prefork)mod_wsgi (embedded) 150 1 Apache (worker)mod_wsgi (embedded) 6 25 Apache (prefork) mod_wsgi (daemon) 1 15 Apache (worker) mod_wsgi (daemon) 1 15
  20. 20. Other WSGI serversConfiguration Max Processes Threads FASTCGIflup (prefork) 50 1 FASTCGIflup (threaded) 1 5 gunicorn 1 1 uWSGI 1 1 tornado 1 1
  21. 21. Less than fair 150 135 Apache (prefork) 120 mod_wsgi (embedded) 105 FASTCGI 90 flup (prefork) 75 Apache (prefork/worker)Processes 60 mod_wsgi (daemon) 45 Apache (worker) 30 mod_wsgi (embedded) 15 0 0 15 30 45 60 75 90 105 120 135 150 Threads
  22. 22. What to use?✴Number of overall threads dictated by: ✴Number of concurrent users. ✴Response time for requests.✴Processes preferred over threads, but: ✴Restricted by amount of memory. ✴Choice influenced by number of processors.
  23. 23. Thread utilisation 1 second 6 5Threads 4 3 2 1
  24. 24. Request backlog Backlog occurred and queue time increased to 750 ms150ms 60 requests Thread utilisation jumped from per second 2.5 to 7.5 and maxed out at 9
  25. 25. Processes are better Backlog only started at higher throughput and queue time mostly under 100ms100ms Thread utilisation only jumped from 75 requests 2.5 to 7.5 at higher throughput per second and didnt actually reach 9
  26. 26. CPU bound Bulk of time is from doingthings within the process itself
  27. 27. I/O wait Waiting on responses frombackend services a significant proportion of time
  28. 28. Long running requests✴Complex calculations.✴Slow backend services.✴Large file uploads.✴Large responses.✴Slow HTTP clients.
  29. 29. Varying request timesAverage: 1385 msMinimum: 4.7 msMaximum: 20184 msStd Dev: 3896 ms
  30. 30. Performance breakdownWhy is creating the connectionto PostgreSQL taking up 40% of overall response time
  31. 31. Slow HTTP clients✴Add nginx as a front end to the WSGI server.✴Brings the following benefits to the WSGI server. ✴Isolation from slow clients. ✴No need to handle keep alive in the WSGI server. ✴Can offload serving of static files. ✴Can use X-Accel-Redirect for dynamically generated files.
  32. 32. Request funnelling nginxfront endApacheworkersmod_wsgidaemons
  33. 33. Complete overload
  34. 34. Forced restarts✴Triggers for restarts: ✴Manual restart to fix issues/configuration. ✴Maximum number of requests reached. ✴Reloading of new application code. ✴Individual requests block/timeout.✴Restarts can make things worse.
  35. 35. Auto scaling✴Apache/mod_wsgi embedded mode. ✴Apache prefork MPM defaults. ✴Initial 1 / Maximum 150 ✴Apache worker MPM defaults. ✴Initial 2 / Maximum 6✴Auto scaling can make things worse.
  36. 36. Pre load everything✴Start maximum processes up front.✴Pre load your web application when the process starts and not lazily loaded on the first request.✴Keep processes persistent in memory and avoid unnecessary restarts.
  37. 37. Horizontal scaling✴Using more servers is fine.✴Load balance across dedicated hosts.✴Or add additional hosts as required.✴Ensure though that if adding more hosts that you have preloaded the web application before directing traffic to it.
  38. 38. Monitoring is key✴Treat your server as a black box and you will never know what is going on inside.
  39. 39. Server monitoring✴Open source tools. ✴Monit ✴Munin ✴Cacti ✴Nagios
  40. 40. Python web tools✴Django debug toolbar. ✴Only useful for debugging a single request in a development setting.✴Sentry. ✴Useful for capturing runtime errors, but performance issues dont generate exceptions.
  41. 41. New Relic APM
  42. 42. Apache/mod_wsgi
  43. 43. Summing up✴Use benchmarks to explore a specific system, not to compare different systems.✴Dont trust the defaults of any server, you need to tune it for your web application.✴Monitor your live production systems.✴New Relic for really deep introspection.
  44. 44. Try New Relic✴ Graham.Dumpleton@gmail.com✴ http://www.slideshare.net/GrahamDumpleton✴ Find out more about New Relic: ✴ http://newrelic.com✴ Extended Pro Trial for PyCon attendees: ✴ http://newrelic.com/30✴ Come work for New Relic: ✴ http://newrelic.com/about/jobs

×