PyCon AU 2012 - Debugging Live Python Web Applications

Debugging Live Python Web Applications Graham Dumpleton / Amjith Ramanujam PyCon AU - August 2012

Follow along. http://www.slideshare.net/GrahamDumpleton

What is debugging? Debugging is a methodical process of ﬁnding and reducing the number of bugs, or defects, in a computer program or a piece of electronic hardware, thus making it behave as expected. http://en.wikipedia.org/wiki/Debugging

Common types of computer bugs. Arithmetic bugs Logic bugs Syntax bugs Multi-threading Resource bugs Interfacing bugs programming bugs Performance bugs Teamworking bugs http://en.wikipedia.org/wiki/Software_bug

Things we want to avoid.• Crashing the whole web site.• Corrupt all your customer data.• Make you customer data visible to everyone.• Loose your company lots of money.• Loose your own job because you did something stupid.• Cause all your work mates to loose their jobs as well.• Getting what you did posted on Slashdot.

Managing risk.• Use software to restrict what you can do.• Script changes and procedures to avoid errors.• Test what you are going to do on a separate system.• Develop and document contingency plans.

Passive monitoring.• Collection of log ﬁle information.• Collection of details about Python exceptions.• Collection of performance data for the server host.• Collection of performance data for the web server.• Collection of performance data for the web application.

Log ﬁle collation and analysis.• Open Source • logstash (http://logstash.net) • graylog2 (http://www.graylog2.org)• Commercial Services • Loggly (http://www.loggly.com) • Splunk (http://www.splunk.com) • LogLogic (http://www/loglogic.com)

Recording Python exceptions.• Open Source • Sentry (http://pypi.python.org/pypi/sentry) - Also as paid service.• Commercial Services • New Relic (http://newrelic.com) - Pro feature.

Server monitoring.• Open Source • Monit (http://mmonit.com) • Munin (http://munin-monitoring.org) • Cacti (http://www.cacti.net) • Nagios (http://www.nagios.org)• Commercial Services • New Relic (http://newrelic.com) - Free feature.

Application performance monitoring.• Commercial Services • New Relic (http://newrelic.com) - Lite (Free), Standard and Pro subscriptions.

Your web site is slow.

Web page performance analysis.• Online services. • YSlow (http://developer.yahoo.com/yslow/) • Google PageSpeed (https://developers.google.com/speed/pagespeed/) • WebPageTest (http://www.webpagetest.org/)• Browser plugins. • YSlow for FireFox (https://addons.mozilla.org/en-US/ﬁrefox/addon/yslow/) • FireBug (http://getﬁrebug.com/)

World Wide Web Consortium.• Resource timing speciﬁcation. • W3 Resource Timing Speciﬁcation (http://www.w3.org/TR/resource-timing/)

Application performance analysis.

Slow transaction summary.

Slow transaction details.

Unknown consumers of time. ????

Instrumentation via code change. import newrelic.agent class _Database(UserDict.DictMixin): @newrelic.agent.function_trace() def _commit(self): ... @newrelic.agent.function_trace() def open(file, flag=None, mode=0666): ...

Instrumentation via conﬁguration. [newrelic] transaction_tracer.function_trace = dumbdbm:open dumbdbm:_Database._commit

Instrumentation by monkey patching.[import-hook:dumbdbm]enabled = trueexecute = dumbdbm_instrumentation:instrument# dumbdbm_instrument.pyfrom newrelic.api.function_trace import wrap_function_tracedef instrument(module): wrap_function_trace(module, open) wrap_function_trace(module, _Database._commit)

Thread sampling.

Profiling tools.• Thread sampling. • plop (http://tech.dropbox.com/?p=272) • statprof (http://pypi.python.org/pypi/statprof/)• Full profiling. • cprofile (http://docs.python.org/library/profile.html) • pytrace (http://pypi.python.org/pypi/pytrace)

Targeted function proﬁling. @function_profile(filename=/tmp/profile.dat, delay=1.0, checkpoint=30) def open(file, flag=None, mode=0666): ...

Controlling what is proﬁled. class FunctionProfile(object): def __init__(self, profile): self.profile = profile def __enter__(self): self.profile.enable() return self def __exit__(self, exc, value, tb): self.profile.disable()

Manual metric collection.• Open Source • metrology - http://metrology.readthedocs.org/en/latest/index.html • mmstats - https://github.com/schmichael/mmstats • pymetrics - https://github.com/jgardner1/Python-Metrics • django-app-metrics - http://pypi.python.org/pypi/django-app-metrics • django-statsd - http://django-statsd.readthedocs.org/en/latest/

Interacting via the browser.• Open Source • Paste Error Middleware - http://pythonpaste.org/modules/exceptions.html • django-debug-toolbar - https://github.com/django-debug-toolbar/django-debug-toolbar/ • Paste Debugger - http://pythonpaste.org/modules/evalexception.html • Flask Debugger - http://werkzeug.pocoo.org/docs/debug/

Application backdoors. import logging import logging.config logging.config.fileConfig("logging.conf") backdoor = logging.config.listen() backdoor.start()

Interactive access.• Embedded interpreter prompt. • eventlet.backdoor - http://eventlet.net/doc/modules/backdoor.html • guppy.heapy.Console - http://guppy-pe.sourceforge.net • twisted.manhole - http://www.lothar.com/tech/twisted/manhole.xhtml• Code injection mechanisms. • pyrasite - http://pyrasite.readthedocs.org/en/latest/index.html• Remote code debuggers. • Komodo IDE - http://www.activestate.com/komodo-ide • PyCharm IDE - http://www.jetbrains.com/pycharm/ • Wing IDE - http://wingware.com/ • PyDev IDE - http://pydev.org/

Introducing ispyd.• Download site. • https://github.com/GrahamDumpleton/wsgi-shell• Aims of the package. • Provide a generic framework for implementing an interactive console. • The commands you can run are targeted at a speciﬁc purpose. • Plugin based so can control what is available and also extendable. • Remotely accessible and execution of commands scriptable.

Connecting to processes. $ ispy ispyd.ini (ispyd) servers 1: (1, /tmp/ispyd-14905.sock) 2: (1, /tmp/ispyd-14906.sock) 3: (1, /tmp/ispyd-14907.sock) (ispyd) connect 1 (ispyd:ll345) plugins [debugger, process, python, wsgi]

Executing commands.(ispyd:ll345) shell process(process:ll345) helpDocumented commands (type help <topic>):========================================cwd egid euid exit gid help pid prompt uid(process:ll345) cwd/Users/graham

Power users.(ispyd:ll345) shell python(python:ll345) consolePython 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)[GCC 4.2.1 (Apple Inc. build 5646)] on darwinType "help", "copyright", "credits" or "license"for more information.(EmbeddedConsole)>>> import os>>> os.getcwd()/Users/graham>>> exit()

Post-mortem debugging.(ispyd:ll345) shell debugger(debugger:11345) insert __main__:function(debugger:11345) tracebacks{__main__:function: <traceback object at0x1013a11b8>}(debugger:11345) debug __main__:function> /Users/graham/wsgi.py(15)function()-> raise RuntimeError(xxx)(Pdb) dir()[](Pdb) __file__wsgi.py

Extending what is monitored.(ispyd:ll345) shell newrelic(newrelic:ll345) function_trace dumbdbm:open(newrelic:ll345) function_trace dumbdbm:_Database._commit

Capacity Analysis Rolling server restart.

Was the customer impacted?

Stuck requests and backlog.

Active requests. (ispyd:ll345) shell requests (debugger:11345) requests ==== 707 ==== thread_id = 140735076232384 start_time = Mon Apr 9 21:49:54 2012 duration = 0.013629 seconds CONTENT_LENGTH = ... File: "wsgi.py", line 25, in <module> application.run(host=0.0.0.0, port=port) ...

Multiprocess web applications. $ ispy --batch - ispyd.ini << EOF prompt off shell requests requests exit shell newrelic function_trace dumbdbm:open function_trace dumbdbm:_Database._commit exit exit EOF

Creating plugins.import psutilclass Shell(object): name = psutil def do_num_cpus(self, line): print >> self.stdout, psutil.NUM_CPUS def do_cpu_times(self, line): print >> self.stdout, psutil.cpu_times() def do_virtual_memory(self, line): print >> self.stdout, psutil.virtual_memory() def do_swap_memory(self, line): print >> self.stdout, psutil.swap_memory()

Ideas for third party plugins.• Memory. • Process memory usage. • Statistics on objects in use (heapy). • State of the garbage collector.• Profiling. • Initiate sampled profiling for selected functions.• Django. • Current configuration. • Details of loaded applications. • Details of registered middleware. • Details of template libraries. • Testing URLs against URL resolver. • Statistics on cache usage.

What am I trying to say?• Use monitoring so you know when problems arise.• One tool alone is not going to provide everything.• Use complimentary tools to get a full picture.• Build in mechanisms that allow deeper debugging.• Treat debugging like any other deﬁned process.

New Relic 30 Day Free Pro Trail http://newrelic.com/30 Graham.Dumpleton@gmail.com @GrahamDumpleton

http://lanyrd.com	165
https://twitter.com	41
http://blaze.apphb.com	7
https://si0.twimg.com	3
https://twimg0-a.akamaihd.net	2
http://a0.twimg.com	2

PyCon AU 2012 - Debugging Live Python Web Applications

by Graham Dumpleton on Aug 18, 2012

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 220

Statistics

PyCon AU 2012 - Debugging Live Python Web Applications Presentation Transcript