Logging in Python for large applications

2,265 views
2,159 views

Published on

Logging in Python for large scale applications.

Published in: Technology, Business
1 Comment
6 Likes
Statistics
Notes
No Downloads
Views
Total views
2,265
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
37
Comments
1
Likes
6
Embeds 0
No embeds

No notes for slide

Logging in Python for large applications

  1. 1. Why use logging Basics When to use what? Logbook Logging in Python For complex applications Fayaz Yusuf Khan 20 December 2013 Error reporting The end
  2. 2. Why use logging Basics When to use what? Logbook Why use logging? Error reporting The end
  3. 3. Why use logging Basics When to use what? Logbook Why use logging? • All errors happen in runtime. Error reporting The end
  4. 4. Why use logging Basics When to use what? Logbook Why use logging? • All errors happen in runtime. • Testing can never be 100% Error reporting The end
  5. 5. Why use logging Basics When to use what? Logbook Error reporting import logging logging . warning ( ’ Watch out ! ’) # will print a message logging . info ( ’I told you so ’) # will not print a n y t h i n g The end
  6. 6. Why use logging Basics When to use what? Logbook Error reporting import logging logging . warning ( ’ Watch out ! ’) # will print a message logging . info ( ’I told you so ’) # will not print a n y t h i n g Output: WARNING:root:Watch out! The end
  7. 7. Why use logging Basics When to use what? Logbook Error reporting Logging to a file import logging logging . basicConfig ( filename = ’ example . log ’ , level = logging . DEBUG ) logging . debug ( ’ This message should go to the log file ’) logging . info ( ’ So should this ’) logging . warning ( ’ And this , too ’) The end
  8. 8. Why use logging Basics When to use what? Logbook Error reporting Logging to a file import logging logging . basicConfig ( filename = ’ example . log ’ , level = logging . DEBUG ) logging . debug ( ’ This message should go to the log file ’) logging . info ( ’ So should this ’) logging . warning ( ’ And this , too ’) Output: DEBUG:root:This message should go to the log file INFO:root:So should this WARNING:root:And this, too The end
  9. 9. Why use logging Basics When to use what? Logbook Initializing logger = logging . getLogger ( __name__ ) Error reporting The end
  10. 10. Why use logging Basics When to use what? Logbook Error reporting The end print() Display console output for ordinary usage of a command line script or program
  11. 11. Why use logging Basics When to use what? Logbook Error reporting logging.info() Report events that occur during normal operation of a program (e.g. for status monitoring or fault investigation) The end
  12. 12. Why use logging Basics When to use what? Logbook Error reporting logging.debug() Very detailed output for diagnostic purposes The end
  13. 13. Why use logging Basics When to use what? Logbook Error reporting warnings.warn() In library code if the issue is avoidable and the client application should be modified to eliminate the warning. The end
  14. 14. Why use logging Basics When to use what? Logbook Error reporting logging.warning() If there is nothing the client application can do about the situation, but the event should still be noted The end
  15. 15. Why use logging Basics When to use what? Logbook Error reporting Report an error regarding a particular runtime event The end
  16. 16. Why use logging Basics When to use what? Logbook Error reporting Raise an exception Report an error regarding a particular runtime event The end
  17. 17. Why use logging Basics When to use what? Logbook Error reporting logging.error() or logging.exception() Report suppression of an error without raising an exception (e.g. error handler in a long-running server process) The end
  18. 18. Why use logging Basics When to use what? Logbook Error reporting The end logging.critical() A serious error, indicating that the program itself may be unable to continue running.
  19. 19. Why use logging Basics When to use what? Logbook Error reporting Logbook pythonhosted.org/Logbook Logbook is a logging sytem for Python that replaces the standard library’s logging module. The end
  20. 20. Why use logging Basics When to use what? Logbook Error reporting >>> from logbook import Logger >>> log = Logger ( ’ Logbook ’) >>> log . info ( ’ Hello , World ! ’) [2010 -07 -23 16:34] INFO : Logbook : Hello , World ! The end
  21. 21. Why use logging Basics When to use what? Logbook Error reporting Context managers everywhere! from logbook import SyslogHandler error_handler = SyslogHandler ( ’ logbook example ’ , level = ’ ERROR ’) with error_handler . a p p l i c a t i o n b o u n d (): # wh a t e v e r is e x e c u t e d here and an error is logged to # the error handler ... The end
  22. 22. Why use logging Basics When to use what? Logbook Error reporting Getting your libraries and old code to behave from logbook . compat import r e d i r ec t _ l o g g i n g r e d i r e ct _ l o g g i n g () The end
  23. 23. Why use logging Basics When to use what? Logbook Error reporting Injecting context import os from logbook import Processor def inject_cwd ( record ): record . extra [ ’ cwd ’] = os . getcwd () with my_handler . a p p l i c at i o n b o u n d (): with Processor ( inject_cwd ). a p p l i c a t i o n b o u n d (): # e v e r y t h i n g logged here will have the current # working d i r e c t o r y in the log record . ... The end
  24. 24. Why use logging Basics When to use what? Logbook Error reporting Formatting in Logging import logging logging . basicConfig ( format = ’ %( levelname ) s :%( message ) s ’ , level = logging . DEBUG ) The end
  25. 25. Why use logging Basics When to use what? Logbook Error reporting Formatting in Logbook >>> from logbook import StderrHandler >>> handler = StderrHandler ( ... format_string = ’{ record . channel }: { record . message } ’) >>> handler . formatter < logbook . handlers . St ri n gF or ma t te r object at 0 x100641b90 > The end
  26. 26. Why use logging Basics When to use what? Logbook Custom formatters >>> def my_formatter ( record , handler ): ... return record . message ... >>> handler . formatter = my_formatter Error reporting The end
  27. 27. Why use logging Basics When to use what? Logbook Custom formatters >>> def my_formatter ( record , handler ): ... return record . message ... >>> handler . formatter = my_formatter Tip: Or use Jinja! Error reporting The end
  28. 28. Why use logging Basics When to use what? Logbook Error reporting The end Extra fields handler = StderrHandler ( format_string = ’{ record . channel }: { record . message ) [{ record . extra [ cwd ]}] ’)
  29. 29. Why use logging Basics When to use what? Logbook Redis import logbook from logbook . queues import RedisHandler handler = RedisHandler () l = logbook . Logger () with handler : l . info ( ’ Your log message ’) Error reporting The end
  30. 30. Why use logging Basics When to use what? Logbook Error reporting RabbitMQ Handler handler = R a bb it MQ H an dl er ( ’ amqp :// guest : gu e st @l oc a lh os t // ’ , queue = ’ my_log ’) Subscriber with target_h andler : subscriber = R a b b i t M Q S u b s c r i b e r ( ’ amqp :// guest : gu e st @l oc a lh os t // ’ , queue = ’ my_log ’) subscriber . d i s p a t ch _ f o r e v e r () The end
  31. 31. Why use logging Basics When to use what? Logbook ZeroMQ Don’t use it. (It’s broken) Error reporting The end
  32. 32. Why use logging Basics When to use what? Logbook Error reporting The end Minimize logging >>> from logbook import FingersCrossedHandler , StderrHandler >>> handler = F i n g e r s C r o s s e d H a n d l e r ( StderrHandler ()) >>> with handler . a p p l i c a t i o n b o u n d (): ... logbook . info ( ’ Hello ’) ... ... >>> handler = F i n g e r s C r o s s e d H a n d l e r ( StderrHandler ()) >>> with handler . a p p l i c a t i o n b o u n d (): ... logbook . info ( ’ Hello ’) ... logbook . error ( ’ Bye ’) ... ... [2013 -12 -16 11:12] INFO : Generic : Hello [2013 -12 -16 11:12] ERROR : Generic : Bye
  33. 33. Why use logging Basics When to use what? Logbook Error reporting TestHandler >>> from logbook import TestHandler , Logger >>> logger = Logger ( ’ Testing ’) >>> handler = TestHandler () >>> handler . push_thread () >>> logger . warn ( ’ Hello World ’) >>> handler . records [ < logbook . base . LogRecord object at 0 x100640cd0 >] >>> handler . f o r m a t t e d _ r e c o r d s [ u ’[ WARNING ] Testing : Hello World ’] The end
  34. 34. Why use logging Basics When to use what? Logbook Github github.com/mitsuhiko/logbook Error reporting The end
  35. 35. Why use logging Basics When to use what? Logbook Error reporting Error reporting The end
  36. 36. Why use logging Basics When to use what? Logbook Error reporting Emails? Error reporting The end
  37. 37. Why use logging Basics When to use what? Logbook Error reporting Error reporting The end
  38. 38. Why use logging Basics When to use what? Logbook Error reporting The end Airbrake Airbrake is the leading exception reporting service, currently providing error tracking for 50,000 applications with support for 18 programming languages.
  39. 39. Why use logging Basics When to use what? Logbook Sign up airbrake.io Error reporting The end
  40. 40. Why use logging Basics When to use what? Logbook Error reporting Flask from flask import Flask , request from airbrake import A i r b r a k e E r r o r H a n d l e r import gevent @app . errorhandler (500) def interna l_error ( error ): if app . config [ ’ E X C E P T I O N _ L O G G I N G ’ ]: handler = A i r b r a k e E r r o r H a n d l e r ( api_key = app . config [ ’ A I R B R E A K _ A P I _ K E Y ’] , env_name = ENV , request_url = request . url , request_path = request . path , requ est_meth od = request . method , request_args = request . args , r eq ue st _ he ad er s = request . headers ) gevent . spawn ( handler . emit , error ) The end
  41. 41. Why use logging Basics When to use what? Logbook Error reporting Django AIRBRAKE = { ’ API_KEY ’: ’ y o u r _ a p i _ k e y _ h e r e ’ , ’ USE_SSL ’: True , ’ TIMEOUT ’: 5 , ’ ENVIRONMENT ’: ’ u n i q u e _ n a m e _ f o r _ e n v i r o n m e n t ’ , } The end
  42. 42. Why use logging Basics When to use what? Logbook Errbit github.com/errbit/errbit Error reporting The end
  43. 43. Why use logging Basics When to use what? Logbook Error reporting The end Sentry getsentry.com Sentry is a realtime event logging and aggregation platform. At its core it specializes in monitoring errors and extracting all the information needed to do a proper post-mortem without any of the hassle of the standard user feedback loop.
  44. 44. Why use logging Basics When to use what? Detail Logbook Error reporting The end
  45. 45. Why use logging Basics When to use what? Logbook More detail Error reporting The end
  46. 46. Why use logging Basics When to use what? Logbook Install the server sentry.readthedocs.org Error reporting The end
  47. 47. Why use logging Basics When to use what? Logbook Error reporting Sentry + Logbook raven.readthedocs.org/en/latest/config/logbook.html Install raven pip install raven Your app code will have this at the entry point from raven . handlers . logbook import SentryHandler client = Client (...) sent ry_handl er = SentryHandler ( client ) with sentry_h andler . a p pl i c a t i o n b o u n d (): # e v e r y t h i n g logged here will go to sentry . ... The end
  48. 48. Why use logging Basics When to use what? Logbook Error reporting Pay attention to formatting! Sentry will intelligently group messages if you use proper string formatting. logger . error ( ’ There was some {} error ’ , ’ crazy ’) logger . error ( ’ There was some {} error ’ , ’ fun ’) logger . error ( ’ There was some {} error ’ , 1) The end
  49. 49. Why use logging Basics When to use what? Logbook Tagging logger . error ( ’ Could not do that ’ , tags ={ ’ user email ’: ’ foo@bar ’ }) Error reporting The end
  50. 50. Why use logging Basics When to use what? Logbook Error reporting The end TicketingHandler Create unique tickets for log records and keep track of the number of times these log records were created from logbook import ERROR from logbook . ticketing import T i c k e t i n g H a n d l e r handler = T i c k e t i n g Ha n d l e r ( ’ postgres :// localhost / database ’ , level = ERROR ) with handler : # e v e r y t h i n g in this block and thread will be handled by # the t i c k e t i n g d a t a b a s e handler ...
  51. 51. Why use logging Basics When to use what? Logbook Error reporting Phew! It’s finally over! Reach me at fayaz at dexetra dot com. The end

×