Web applicationWeb application Front end time 0.15 seconds 3.1 seconds
Performance Golden Rule "80-90% of the end-user response time is spent on the frontend. Start there."http://www.stevesouders.com/blog/2012/02/10/ the-performance-golden-rule/
Are benchmarks stupid?http://nichol.as/benchmark-of-python-web-servers
Benchmarks as a tool✴Web server benchmarks are of more value when used as an exploratory tool to understand how a speciﬁc system works, not to compare systems.
What about load tests?✴Hitting a site with extreme load will only show you that it will likely fail under a denial of service attack.✴Your typical web server load test isnt alone going to help you understand how a web server is contributing to it failing.
What should you test?✴You should use a range of purpose built tests to trigger certain scenarios.✴Use the tests to explore corner cases and not just the typical use case.
Environment factors✴Amount of memory available.✴Number of processors available.✴Use of threads vs processes.✴Python global interpreter lock (GIL)
What to use?✴Number of overall threads dictated by: ✴Number of concurrent users. ✴Response time for requests.✴Processes preferred over threads, but: ✴Restricted by amount of memory. ✴Choice inﬂuenced by number of processors.
Thread utilisation 1 second 6 5Threads 4 3 2 1
Request backlog Backlog occurred and queue time increased to 750 ms150ms 60 requests Thread utilisation jumped from per second 2.5 to 7.5 and maxed out at 9
Processes are better Backlog only started at higher throughput and queue time mostly under 100ms100ms Thread utilisation only jumped from 75 requests 2.5 to 7.5 at higher throughput per second and didnt actually reach 9
CPU bound Bulk of time is from doingthings within the process itself
I/O wait Waiting on responses frombackend services a signiﬁcant proportion of time
Performance breakdownWhy is creating the connectionto PostgreSQL taking up 40% of overall response time
Slow HTTP clients✴Add nginx as a front end to the WSGI server.✴Brings the following beneﬁts to the WSGI server. ✴Isolation from slow clients. ✴No need to handle keep alive in the WSGI server. ✴Can ofﬂoad serving of static ﬁles. ✴Can use X-Accel-Redirect for dynamically generated ﬁles.
Forced restarts✴Triggers for restarts: ✴Manual restart to ﬁx issues/conﬁguration. ✴Maximum number of requests reached. ✴Reloading of new application code. ✴Individual requests block/timeout.✴Restarts can make things worse.
Auto scaling✴Apache/mod_wsgi embedded mode. ✴Apache prefork MPM defaults. ✴Initial 1 / Maximum 150 ✴Apache worker MPM defaults. ✴Initial 2 / Maximum 6✴Auto scaling can make things worse.
Pre load everything✴Start maximum processes up front.✴Pre load your web application when the process starts and not lazily loaded on the ﬁrst request.✴Keep processes persistent in memory and avoid unnecessary restarts.
Horizontal scaling✴Using more servers is ﬁne.✴Load balance across dedicated hosts.✴Or add additional hosts as required.✴Ensure though that if adding more hosts that you have preloaded the web application before directing trafﬁc to it.
Monitoring is key✴Treat your server as a black box and you will never know what is going on inside.
Server monitoring✴Open source tools. ✴Monit ✴Munin ✴Cacti ✴Nagios
Python web tools✴Django debug toolbar. ✴Only useful for debugging a single request in a development setting.✴Sentry. ✴Useful for capturing runtime errors, but performance issues dont generate exceptions.
New Relic APM
Summing up✴Use benchmarks to explore a speciﬁc system, not to compare different systems.✴Dont trust the defaults of any server, you need to tune it for your web application.✴Monitor your live production systems.✴New Relic for really deep introspection.
Try New Relic✴ Graham.Dumpleton@gmail.com✴ http://www.slideshare.net/GrahamDumpleton✴ Find out more about New Relic: ✴ http://newrelic.com✴ Extended Pro Trial for PyCon attendees: ✴ http://newrelic.com/30✴ Come work for New Relic: ✴ http://newrelic.com/about/jobs