Search

Monitoring and Debugging your Live Applications

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

1 comments

Comments 1 - 1 of 1 previous next Post a comment

  • + rcoup Robert Coup 2 weeks ago
    There are notes on most of the slides (click Notes instead of Comments underneath) describing roughly what was presented, as well as a paper with full code examples and further details.
Post a comment
Embed Video
Edit your comment Cancel

Notes on slide 1

When we encounter a problem with our live applications (on the web or elsewhere), we often have no opportunity to follow the normal debugging process through to its conclusion. The app needs to get back up, stat! And trying to piece together logs afterwards is hard.

I believe in logging virtually everything - if it helps during development it’ll probably help during debugging. Certainly beats “turning up” or adding logging, restarting the app, and waiting for the problem to happen again.

who logs like this?

What about this? Standard error is good, right? Separates our logging from any real output...

the problem with these is you end up commenting everything just to make it shut the hell up. And then have to uncomment to start talking again.
http://www.flickr.com/photos/larimdame/2575986601

whoa? an actual logging framework? Yes, Python has a powerful builtin logging framework with the logging module, we should use it.

Logging separates the source of the messages from the output. So you can turn up and down the level in configuration (live even), or point logs somewhere else, without ever going back into the code itself.

Python’s logging module gets a real bad rap about the place...
http://www.flickr.com/photos/theeerin/1108095143

Lots of people think it’s based too closely on log4j. How many ways are there to specify log.debug? It’s actually a pretty clean design that works fast.
http://www.flickr.com/photos/lumaxart/2136948489

Others say its too hard. Seriously, 3x lines of code to get file logging going for an entire app is not “too hard”

The first key component of logging is a Logger, which generates log messages. Loggers are named by the developer, usually based on where the code is. The names form a hierarchy using the dots. Then we have the debugging methods at different levels. For the exception() one it’ll just pick up on the current exception and log a traceback all by itself. And all the logging methods accept % formatting via their arguments, so we don’t have to build strings just to chuck them away.

Handlers do something with log messages. They might write it to the console, to a file, send to a web server, post an email, stick it in Syslog or the NT event log, or a multitude of other things. They can filter out which messages they apply to, so we can log everything to a file as well as email just the critical errors.

The last one is a formatter. What are our log messages going to look like? The formatter has access to the code location where the log message came from, as well as the process, thread, the time, and so on.

So lets start with an example. We have our “machine” application, which wants to log different things to various places.

So in our code, we don’t need much. Somewhere we call fileConfig to set things up, pointing to our ini-style config. After that we just get logger objects by name, then call methods on them.

so, we want to send everything to a file by default. This bit of config will do that. We set up a format for our log lines. Since the loggers form a hierarchy, the root one is at the top and everything floats up there eventually. And we tell it to use a FileHandler pointing to our log.

Next, startup/shutdown messages to syslog. Anything logged by a logger named machine.startstop.something will end up here. We use a SysLogHandler to get it into syslog for us.

We want to squash the debug & info messages for three libraries: noisy, chatty, loud. We do this by setting propagate=0, which prevents the messages from bubbling up to the root logger. And we have no handler, so they don’t go anywhere else.

Then we want to send core errors (machine.core.anything) by email. We use level=error to filter out just the errors. And then we use a SMTPHandler to log via email.

And that’s prettymuch logging. There are some other tools I’m fond of, since you’re logging everything... Syslog-ng can help pull logs from multiple machines into one place. ack & less are the awesome duo of finding anything in text. And Splunk is a great tool for matching and tracing lots of events across multiple servers.

(DEMO 4-1.py)

The next thing in our debugging arsenal is a remote console. Don’t you wish you could just look inside your app and find out what it’s thinking, rather than hypothesising? Remote consoles certainly beats trial and error, or reconstructing chains of events from log files. We have the technology to just ask it :)
http://www.flickr.com/photos/pasukaru76/3959355664/

Who’s heard of Twisted? The twisted library does lots of magic things, but one of the cool ones is a telnet or ssh server that gives access into a running python environment. You can read and change variables, call methods, whatever you want. (DEMO 2-1.py)

And the code is pretty simple. Basically we create a dictionary object we use as a namespace for where the user ends up. We specify an IP and a port (and encryption keys if you want an ssh shell), and a file with usernames and passwords. that’s it. At least, if you have a Twisted app.

Twisted is an asynchronous framework, it’s single threaded with an event loop, and doesn’t work in the same way most Python code does. But we can run the Twisted part (which does our manhole) in a separate thread, leaving our normal code alone. The only downside? we have to handle concurrency if we’re changing things :) (DEMO 2-2.py)

The 3rd idea i want to talk about is robots. We can review our app’s progress with logs. We can telnet right inside it for hardcore inspection. But we can also get it talking to us via instant messaging.
http://saiogaman.deviantart.com/art/Danbo-Wallpaper-107237965

everybody runs IM of some sort, its already open on the desktop, it just works. Our apps can say to us, in real time: “hey, Rob, I’m not happy”. Plus, the marketing guy down the hall might like to get pinged if someone orders 100 copies of our awesome software. Bots can do this.

Among its arsenal of networking awesomeness, twisted also does XMPP (also known as Jabber). It’ll also do IRC and some other stuff, but we’ll focus on XMPP for now. Basically it’s an asynchronous protocol for passing bits of XML around. No polling required. There’s a library called Wokkel which makes XMPP even easier than Words does. (DEMO 3-1.py)

This is the talk-ey part of the code for that example. It gets a message object in, then composes and sends a reply. Do whatever you want in these methods. And your app can call .send() itself anytime to send messages.

And this is basically what I’d like you to take away. Arm yourself with some great tools and debugging and keeping an eye on your apps will be a lot easier. Trial and error sucks, move on!

2 Favorites & 1 Group

Monitoring and Debugging your Live Applications - Presentation Transcript

  1. Monitoring & Debugging Live Applications Rob Coup robert@coup.net.nz
  2. Who am I? • Koordinates • Open data open.org.nz • Geek • Pythonista
  3. Crash! • trace ➭ reproduce ➭ simplify ➭ solve • post-mortems suck
  4. log everything
  5. print "name=", name
  6. print >>sys.stderr, "name=", name
  7. # http://www.flickr.com/photos/larimdame/2575986601
  8. log.debug("name=%s", name)
  9. Separation • where the log messages are generated • where the log output goes to • runtime configuration
  10. http://www.flickr.com/photos/theeerin/1108095143
  11. “too Java” http://www.flickr.com/photos/lumaxart/2136948489
  12. Too hard? import logging logging.basicConfig(level=logging.DEBUG, file="/tmp/my.log") log = logging.getLogger("somewhere")
  13. Logger • named (foo.bar.baz) • debug() info() warn() error() critical() exception() • % formatting
  14. Handler • does something with log messages • stderr, files, syslog, HTTP, email, … • filtering on logger hierarchy & level
  15. Formatter • formats the log output for a Handler • time, file/line/function, thread, process, …
  16. Logging Example • log everything to /tmp/machine.log • log startup and shutdown events to syslog • quieten some noisy libraries (log warnings & errors only) • email errors in machine.core to admin@example.com
  17. In the code logging.config.fileConfig("machine-logging.ini") L = logging.getLogger("machine.startstop.start") L.info("Starting the machine...") L = logging.getLogger("machine.core.reactor") L.error("Meltdown in Sector 7-G!")
  18. Logging Example # our normal formatter [formatter_normal] format=%(asctime)s machine[%(process)d]: %(name)s %(module)s (%(lineno)d): %(message)s # root logger, send everything to the file log [logger_root] level=NOTSET handlers=file # log to a file [handler_file] class=FileHandler formatter=normal args=('/tmp/machine.log','a')
  19. Logging Example # send startup/shutdown notices to syslog [logger_startstop] qualname=machine.startstop handlers=syslog # log to syslog [handler_syslog] class=handlers.SysLogHandler formatter=normal args=('/dev/log',)
  20. Logging Example # ignore messages below WARN for noisy libraries [logger_noisylibs] level=WARN qualname=noisy,chatty,loud propagate=0 handlers=
  21. Logging Example # send machine.core errors to the email handler [logger_core] qualname=machine.core level=ERROR handlers=email # log to email [handler_email] class=handlers.SMTPHandler formatter=email args=('localhost', 'logs@a.com', ['admin@a.com'], 'LOG NOTICE')
  22. Other Log Tools • syslog-ng - centralise • ack - grep on steroids • less - a real pager • Splunk - match & search & filter in bulk
  23. Debuggers • pdb works quite well • even nicer with IPython • pydbgr (Pydb) & WinPDB
  24. Attach to running process • I want “pdb -p 12345” (like gdb) • Alternative: import pdb, signal signal.signal(signal.SIGUSR1, lambda x,y: pdb.set_trace()) • Then: kill -s SIGUSR1 <pid>
  25. Remote Consoles http://www.flickr.com/photos/pasukaru76/3959355664/
  26. Twisted Manhole • Telnet/SSH server • Access to interpreter namespace • Interact with variables, call methods, …
  27. Pretty easy from twisted.conch import manhole_tap manhole = manhole_tap.makeService({ 'namespace': namespace, # listen for Telnet connections on localhost:4777 'telnetPort' : 'tcp:4777:interface=127.0.0.1', 'sshPort' : None, # path to passwd-style file (username:password, plaintext) 'passwd' : 'passwd', }) manhole.setServiceParent(application)
  28. But, uh, Twisted?! • run the Twisted manhole in a thread • leave normal Python code alone • still get remote console • need to handle concurrency!
  29. Bots http://saiogaman.deviantart.com/art/Danbo-Wallpaper-107237965
  30. Why? • interact with our apps from our desktop • have our apps interact with us • great for non-technical users
  31. Twisted Words • for accessing IM services - XMPP, IRC, … • Wokkel makes XMPP even easier
  32. Abuse Bot class TalkProtocol(xmppim.MessageProtocol): def onMessage(self, message): # we've received a message! reply = "nah, you're a " + str(message.body) response = domish.Element((None, "message")) response["to"] = message['from'] response.addElement((None, 'body'), content=reply) self.send(response)
  33. Outcomes • Use logging • Talk to your apps (console, bots) • Have your apps talk to you (bots) • You don’t need to have a Twisted app
  34. Monitoring & Debugging Live Applications Rob Coup robert@coup.net.nz

+ Robert CoupRobert Coup, 2 weeks ago

custom

150 views, 2 favs, 0 embeds more stats

Some ideas about debugging and monitoring live appl more

More info about this document

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Go to text version

  • Total Views 150
    • 150 on SlideShare
    • 0 from embeds
  • Comments 1
  • Favorites 2
  • Downloads 8
Most viewed embeds

more

All embeds

less

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

Cancel
File a copyright complaint
Having problems? Go to our helpdesk?

Categories

Tags

more

Groups / Events

-->
Search
RSS Feed
What's new?

© 2009 SlideShare Inc. All Rights Reserved