Your SlideShare is downloading. ×
Monitoring and Debugging your Live Applications
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Monitoring and Debugging your Live Applications


Published on

Some ideas about debugging and monitoring live applications: logging, remote-shells using Twisted (even in non-twisted apps), python debuggers, and creating IM bots so your apps can talk to you. …

Some ideas about debugging and monitoring live applications: logging, remote-shells using Twisted (even in non-twisted apps), python debuggers, and creating IM bots so your apps can talk to you.

Presented at Kiwi Pycon 2009

Published in: Technology
1 Comment
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • When we encounter a problem with our live applications (on the web or elsewhere), we often have no opportunity to follow the normal debugging process through to its conclusion. The app needs to get back up, stat! And trying to piece together logs afterwards is hard.
  • I believe in logging virtually everything - if it helps during development it’ll probably help during debugging. Certainly beats “turning up” or adding logging, restarting the app, and waiting for the problem to happen again.
  • who logs like this?
  • What about this? Standard error is good, right? Separates our logging from any real output...
  • the problem with these is you end up commenting everything just to make it shut the hell up. And then have to uncomment to start talking again.
  • whoa? an actual logging framework? Yes, Python has a powerful builtin logging framework with the logging module, we should use it.
  • Logging separates the source of the messages from the output. So you can turn up and down the level in configuration (live even), or point logs somewhere else, without ever going back into the code itself.
  • Python’s logging module gets a real bad rap about the place...
  • Lots of people think it’s based too closely on log4j. How many ways are there to specify log.debug? It’s actually a pretty clean design that works fast.
  • Others say its too hard. Seriously, 3x lines of code to get file logging going for an entire app is not “too hard”
  • The first key component of logging is a Logger, which generates log messages. Loggers are named by the developer, usually based on where the code is. The names form a hierarchy using the dots. Then we have the debugging methods at different levels. For the exception() one it’ll just pick up on the current exception and log a traceback all by itself. And all the logging methods accept % formatting via their arguments, so we don’t have to build strings just to chuck them away.
  • Handlers do something with log messages. They might write it to the console, to a file, send to a web server, post an email, stick it in Syslog or the NT event log, or a multitude of other things. They can filter out which messages they apply to, so we can log everything to a file as well as email just the critical errors.
  • The last one is a formatter. What are our log messages going to look like? The formatter has access to the code location where the log message came from, as well as the process, thread, the time, and so on.
  • So lets start with an example. We have our “machine” application, which wants to log different things to various places.
  • So in our code, we don’t need much. Somewhere we call fileConfig to set things up, pointing to our ini-style config. After that we just get logger objects by name, then call methods on them.
  • so, we want to send everything to a file by default. This bit of config will do that. We set up a format for our log lines. Since the loggers form a hierarchy, the root one is at the top and everything floats up there eventually. And we tell it to use a FileHandler pointing to our log.
  • Next, startup/shutdown messages to syslog. Anything logged by a logger named machine.startstop.something will end up here. We use a SysLogHandler to get it into syslog for us.
  • We want to squash the debug & info messages for three libraries: noisy, chatty, loud. We do this by setting propagate=0, which prevents the messages from bubbling up to the root logger. And we have no handler, so they don’t go anywhere else.
  • Then we want to send core errors (machine.core.anything) by email. We use level=error to filter out just the errors. And then we use a SMTPHandler to log via email.
  • And that’s prettymuch logging. There are some other tools I’m fond of, since you’re logging everything... Syslog-ng can help pull logs from multiple machines into one place. ack & less are the awesome duo of finding anything in text. And Splunk is a great tool for matching and tracing lots of events across multiple servers.
  • (DEMO
  • The next thing in our debugging arsenal is a remote console. Don’t you wish you could just look inside your app and find out what it’s thinking, rather than hypothesising? Remote consoles certainly beats trial and error, or reconstructing chains of events from log files. We have the technology to just ask it :)
  • Who’s heard of Twisted? The twisted library does lots of magic things, but one of the cool ones is a telnet or ssh server that gives access into a running python environment. You can read and change variables, call methods, whatever you want. (DEMO
  • And the code is pretty simple. Basically we create a dictionary object we use as a namespace for where the user ends up. We specify an IP and a port (and encryption keys if you want an ssh shell), and a file with usernames and passwords. that’s it. At least, if you have a Twisted app.
  • Twisted is an asynchronous framework, it’s single threaded with an event loop, and doesn’t work in the same way most Python code does. But we can run the Twisted part (which does our manhole) in a separate thread, leaving our normal code alone. The only downside? we have to handle concurrency if we’re changing things :) (DEMO
  • The 3rd idea i want to talk about is robots. We can review our app’s progress with logs. We can telnet right inside it for hardcore inspection. But we can also get it talking to us via instant messaging.
  • everybody runs IM of some sort, its already open on the desktop, it just works. Our apps can say to us, in real time: “hey, Rob, I’m not happy”. Plus, the marketing guy down the hall might like to get pinged if someone orders 100 copies of our awesome software. Bots can do this.
  • Among its arsenal of networking awesomeness, twisted also does XMPP (also known as Jabber). It’ll also do IRC and some other stuff, but we’ll focus on XMPP for now. Basically it’s an asynchronous protocol for passing bits of XML around. No polling required. There’s a library called Wokkel which makes XMPP even easier than Words does. (DEMO
  • This is the talk-ey part of the code for that example. It gets a message object in, then composes and sends a reply. Do whatever you want in these methods. And your app can call .send() itself anytime to send messages.
  • And this is basically what I’d like you to take away. Arm yourself with some great tools and debugging and keeping an eye on your apps will be a lot easier. Trial and error sucks, move on!
  • Transcript

    • 1. Monitoring & Debugging Live Applications Rob Coup
    • 2. Who am I? • Koordinates • Open data • Geek • Pythonista
    • 3. Crash! • trace ➭ reproduce ➭ simplify ➭ solve • post-mortems suck
    • 4. log everything
    • 5. print "name=", name
    • 6. print >>sys.stderr, "name=", name
    • 7. #
    • 8. log.debug("name=%s", name)
    • 9. Separation • where the log messages are generated • where the log output goes to • runtime configuration
    • 10.
    • 11. “too Java”
    • 12. Too hard? import logging logging.basicConfig(level=logging.DEBUG, file="/tmp/my.log") log = logging.getLogger("somewhere")
    • 13. Logger • named ( • debug() info() warn() error() critical() exception() • % formatting
    • 14. Handler • does something with log messages • stderr, files, syslog, HTTP, email, … • filtering on logger hierarchy & level
    • 15. Formatter • formats the log output for a Handler • time, file/line/function, thread, process, …
    • 16. Logging Example • log everything to /tmp/machine.log • log startup and shutdown events to syslog • quieten some noisy libraries (log warnings & errors only) • email errors in machine.core to
    • 17. In the code logging.config.fileConfig("machine-logging.ini") L = logging.getLogger("machine.startstop.start")"Starting the machine...") L = logging.getLogger("machine.core.reactor") L.error("Meltdown in Sector 7-G!")
    • 18. Logging Example # our normal formatter [formatter_normal] format=%(asctime)s machine[%(process)d]: %(name)s %(module)s (%(lineno)d): %(message)s # root logger, send everything to the file log [logger_root] level=NOTSET handlers=file # log to a file [handler_file] class=FileHandler formatter=normal args=('/tmp/machine.log','a')
    • 19. Logging Example # send startup/shutdown notices to syslog [logger_startstop] qualname=machine.startstop handlers=syslog # log to syslog [handler_syslog] class=handlers.SysLogHandler formatter=normal args=('/dev/log',)
    • 20. Logging Example # ignore messages below WARN for noisy libraries [logger_noisylibs] level=WARN qualname=noisy,chatty,loud propagate=0 handlers=
    • 21. Logging Example # send machine.core errors to the email handler [logger_core] qualname=machine.core level=ERROR handlers=email # log to email [handler_email] class=handlers.SMTPHandler formatter=email args=('localhost', '', [''], 'LOG NOTICE')
    • 22. Other Log Tools • syslog-ng - centralise • ack - grep on steroids • less - a real pager • Splunk - match & search & filter in bulk
    • 23. Debuggers • pdb works quite well • even nicer with IPython • pydbgr (Pydb) & WinPDB
    • 24. Attach to running process • I want “pdb -p 12345” (like gdb) • Alternative: import pdb, signal signal.signal(signal.SIGUSR1, lambda x,y: pdb.set_trace()) • Then: kill -s SIGUSR1 <pid>
    • 25. Remote Consoles
    • 26. Twisted Manhole • Telnet/SSH server • Access to interpreter namespace • Interact with variables, call methods, …
    • 27. Pretty easy from twisted.conch import manhole_tap manhole = manhole_tap.makeService({ 'namespace': namespace, # listen for Telnet connections on localhost:4777 'telnetPort' : 'tcp:4777:interface=', 'sshPort' : None, # path to passwd-style file (username:password, plaintext) 'passwd' : 'passwd', }) manhole.setServiceParent(application)
    • 28. But, uh, Twisted?! • run the Twisted manhole in a thread • leave normal Python code alone • still get remote console • need to handle concurrency!
    • 29. Bots
    • 30. Why? • interact with our apps from our desktop • have our apps interact with us • great for non-technical users
    • 31. Twisted Words • for accessing IM services - XMPP, IRC, … • Wokkel makes XMPP even easier
    • 32. Abuse Bot class TalkProtocol(xmppim.MessageProtocol): def onMessage(self, message): # we've received a message! reply = "nah, you're a " + str(message.body) response = domish.Element((None, "message")) response["to"] = message['from'] response.addElement((None, 'body'), content=reply) self.send(response)
    • 33. Outcomes • Use logging • Talk to your apps (console, bots) • Have your apps talk to you (bots) • You don’t need to have a Twisted app
    • 34. Monitoring & Debugging Live Applications Rob Coup