Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Hidden Treasures      of the Standard Library      Doug Hellmann      PyCon      February, 2011Sunday, March 20, 2011
Python Module of the Week:      “I read the docs, so you don’t have to.”Sunday, March 20, 2011
Read This!      Pre-order nowSunday, March 20, 2011
Goals   Usable TipsSunday, March 20, 2011
Look Around   Change FocusSunday, March 20, 2011
Look Around   100+ ModulesSunday, March 20, 2011
Full Name    User Name                Email           Doug Hellmann       dhellmann   doug.hellmann@gmail.com            G...
Test Database             1     #!/bin/sh             2             3     rm -f test.db             4             5     sq...
Test Database             1     #!/bin/sh             2             3     rm -f test.db             4             5     sq...
Database Contents       $ ./hidden_stdlib/csv_mkdb.sh       $ sqlite3 -noheader test.db select * from users       Doug Hel...
Database Contents       $ ./hidden_stdlib/csv_mkdb.sh       $ sqlite3 -noheader test.db select * from users       Doug Hel...
csv Dialect              9 csv.register_dialect(pipes, delimiter=|)             10             11 reader = csv.reader(sys....
csv Dialect              9 csv.register_dialect(pipes, delimiter=|)             10             11 reader = csv.reader(sys....
csv Dialect              9 csv.register_dialect(pipes, delimiter=|)             10             11 reader = csv.reader(sys....
Database Contents       $ ./hidden_stdlib/csv_mkdb.sh       $ sqlite3 -noheader test.db select * from users       Doug Hel...
Database Contents       $ ./hidden_stdlib/csv_mkdb.sh       $ sqlite3 -noheader test.db select * from users       Doug Hel...
Custom Column Types   sqlite3Sunday, March 20, 2011
Custom SQLite Column Types             42          # Insert the objects into the database             43          print In...
Custom SQLite Column Types             42          # Insert the objects into the database             43          print In...
Custom SQLite Column Types             42          # Insert the objects into the database             43          print In...
Custom SQLite Column Types              9 class MyObj(object):             10     def __init__(self, arg):             11 ...
Custom SQLite Column Types             17     def adapter_func(obj):             18         """memory -> storage"""       ...
Custom SQLite Column Types             17     def adapter_func(obj):             18         """memory -> storage"""       ...
Custom SQLite Column Types             17     def adapter_func(obj):             18         """memory -> storage"""       ...
Custom SQLite Column Types             17     def adapter_func(obj):             18         """memory -> storage"""       ...
Custom SQLite Column Types             30 with sqlite3.connect(             31          type_demo.db,             32      ...
Custom SQLite Column Types             30 with sqlite3.connect(             31          type_demo.db,             32      ...
Custom SQLite Column Types       $ python hidden_stdlib/sqlite3_custom_type.py       Inserting:       Saving: MyObj(this i...
Custom SQLite Column Types       $ python hidden_stdlib/sqlite3_custom_type.py       Inserting:       Saving: MyObj(this i...
Custom SQLite Column Types       $ python hidden_stdlib/sqlite3_custom_type.py       Inserting:       Saving: MyObj(this i...
Signed Serialized Data   hmacSunday, March 20, 2011
hmac             11     message = The message             12     encoded_message = pickle.dumps(message)             13   ...
hmac             11     message = The message             12     encoded_message = pickle.dumps(message)             13   ...
hmac             11     message = The message             12     encoded_message = pickle.dumps(message)             13   ...
hmac             26     # "Receive" the message             27     read_signature = buffer.readline().rstrip()            ...
hmac             26     # "Receive" the message             27     read_signature = buffer.readline().rstrip()            ...
hmac             37 if computed_signature == read_signature:             38     print nValid message, processed           ...
hmac             37 if computed_signature == read_signature:             38     print nValid message, processed           ...
hmac       $ python hidden_stdlib/hmac_message_signing.py       Outgoing signature: 3498b6fa42a75dc4603da6f4c67505a2      ...
Serializing Objects   jsonSunday, March 20, 2011
json             10 def convert_to_builtin_type(obj):             11     """object->dictionary"""             12     print...
json             10 def convert_to_builtin_type(obj):             11     """object->dictionary"""             12     print...
json             10 def convert_to_builtin_type(obj):             11     """object->dictionary"""             12     print...
json             10 def convert_to_builtin_type(obj):             11     """object->dictionary"""             12     print...
json       $ python hidden_stdlib/json_dump_default.py       convert_to_builtin_type(<MyObj(instance value goes here)>)   ...
json       $ python hidden_stdlib/json_dump_default.py       convert_to_builtin_type(<MyObj(instance value goes here)>)   ...
json       $ python hidden_stdlib/json_dump_default.py       convert_to_builtin_type(<MyObj(instance value goes here)>)   ...
json       $ python hidden_stdlib/json_dump_default.py       convert_to_builtin_type(<MyObj(instance value goes here)>)   ...
json       $ python hidden_stdlib/json_dump_default.py       convert_to_builtin_type(<MyObj(instance value goes here)>)   ...
json              9 def dict_to_object(d):             10     if __class__ not in d:             11         return d      ...
json              9 def dict_to_object(d):             10     if __class__ not in d:             11         return d      ...
json              9 def dict_to_object(d):             10     if __class__ not in d:             11         return d      ...
json             23          print Loading "%s" from "%s" %              24              (class_name, module_name)        ...
json             23          print Loading "%s" from "%s" %              24              (class_name, module_name)        ...
json             23          print Loading "%s" from "%s" %              24              (class_name, module_name)        ...
json             23          print Loading "%s" from "%s" %              24              (class_name, module_name)        ...
json             35 for encoded_object in [             36                  37     [{"s": "instance value goes here",     ...
json             35 for encoded_object in [             36                  37     [{"s": "instance value goes here",     ...
json             49          try:             50              myobj_instance = json.loads(             51                 ...
json       $ python hidden_stdlib/json_load_object_hook.py       Loading "MyObj" from "json_myobj"       Instantiating wit...
json       $ python hidden_stdlib/json_load_object_hook.py       Loading "MyObj" from "json_myobj"       Instantiating wit...
Error Handling   sys.excepthookSunday, March 20, 2011
sys.excepthook              6 def main():              7     # do some work              8     raise RuntimeError(Helpful ...
sys.excepthook              6 import sys              7              8 def main():              9     try:             10 ...
sys.excepthook              6     import sys              7              8     def quiet_errors(exc_type, exc_value, trace...
sys.excepthook              6     import sys              7              8     def quiet_errors(exc_type, exc_value, trace...
Communicating                                         logging                            with UsersSunday, March 20, 2011
logging   loggersSunday, March 20, 2011
logging   loggersSunday, March 20, 2011
logging             10     # Log verbosely             11     root_logger = logging.getLogger()             12     root_lo...
logging             10     # Log verbosely             11     root_logger = logging.getLogger()             12     root_lo...
logging             10     # Log verbosely             11     root_logger = logging.getLogger()             12     root_lo...
logging             10     # Log verbosely             11     root_logger = logging.getLogger()             12     root_lo...
logging             10     # Log verbosely             11     root_logger = logging.getLogger()             12     root_lo...
logging         21 # Include debug messages when logging to a file         22 file_handler = logging.handlers.RotatingFile...
logging         21 # Include debug messages when logging to a file         22 file_handler = logging.handlers.RotatingFile...
logging         21 # Include debug messages when logging to a file         22 file_handler = logging.handlers.RotatingFile...
logging             30     # Log sample messages with different levels             31     log = logging.getLogger(__name__...
logging             30     # Log sample messages with different levels             31     log = logging.getLogger(__name__...
logging             30     # Log sample messages with different levels             31     log = logging.getLogger(__name__...
logging       $ python hidden_stdlib/logging_example.py       on the console and in the file       simple error message   ...
logging       $ python hidden_stdlib/logging_example.py       on the console and in the file       simple error message   ...
logging       $ python hidden_stdlib/logging_example.py       on the console and in the file       simple error message   ...
logging       $ python hidden_stdlib/logging_example.py       on the console and in the file       simple error message   ...
logging       $ python hidden_stdlib/logging_example.py       on the console and in the file       simple error message   ...
logging   Traceback HandlerSunday, March 20, 2011
logging             36     # Tracebacks should only go to the file             37     traceback_log = logging.getLogger(tr...
logging             36     # Tracebacks should only go to the file             37     traceback_log = logging.getLogger(tr...
logging       $ python hidden_stdlib/logging_example_tracebacks.py       on the console and in the file       simple error...
More Information      • http://docs.python.org/      • http://www.doughellmann.com/PyMOTW/      • https://bitbucket.org/dh...
Image Credits          2. http://www.uottawa.ca.libguides.com/content.php?pid=14825&sid=117658          4. http://www.flick...
Upcoming SlideShare
Loading in …5
×

Hidden Treasures of the Python Standard Library

26,550 views

Published on

The standard library contains many hidden gems that are not widely used, either because they are not publicized enough or because they are deep in a module that programmers haven't had cause to study or use. This presentation covers a few selected topics of this nature in about 25 minutes (leaving time for a couple of questions). Demonstration code is included for every item.

Published in: Technology

Hidden Treasures of the Python Standard Library

  1. Hidden Treasures of the Standard Library Doug Hellmann PyCon February, 2011Sunday, March 20, 2011
  2. Python Module of the Week: “I read the docs, so you don’t have to.”Sunday, March 20, 2011
  3. Read This! Pre-order nowSunday, March 20, 2011
  4. Goals Usable TipsSunday, March 20, 2011
  5. Look Around Change FocusSunday, March 20, 2011
  6. Look Around 100+ ModulesSunday, March 20, 2011
  7. Full Name User Name Email Doug Hellmann dhellmann doug.hellmann@gmail.com Guest account, guest guest@example.com no login Parsing Structured Data csv dialectsSunday, March 20, 2011
  8. Test Database 1 #!/bin/sh 2 3 rm -f test.db 4 5 sqlite3 test.db <<EOF 6 7 CREATE TABLE users ( 8 fullname text, 9 username text, 10 email text 11 ); 12 13 INSERT INTO users (fullname, username, email) 14 VALUES (Doug Hellmann, dhellmann, doug.hellmann@gmail.com); 15 16 INSERT INTO users (fullname, username, email) 17 VALUES (Guest account, no login, guest, guest@example.com); 18 19 EOF 20Sunday, March 20, 2011
  9. Test Database 1 #!/bin/sh 2 3 rm -f test.db 4 5 sqlite3 test.db <<EOF 6 7 CREATE TABLE users ( 8 fullname text, 9 username text, 10 email text 11 ); 12 13 INSERT INTO users (fullname, username, email) 14 VALUES (Doug Hellmann, dhellmann, doug.hellmann@gmail.com); 15 16 INSERT INTO users (fullname, username, email) 17 VALUES (Guest account, no login, guest, guest@example.com); 18 19 EOF 20Sunday, March 20, 2011
  10. Database Contents $ ./hidden_stdlib/csv_mkdb.sh $ sqlite3 -noheader test.db select * from users Doug Hellmann|dhellmann|doug.hellmann@gmail.com Guest account, no login|guest|guest@example.comSunday, March 20, 2011
  11. Database Contents $ ./hidden_stdlib/csv_mkdb.sh $ sqlite3 -noheader test.db select * from users Doug Hellmann|dhellmann|doug.hellmann@gmail.com Guest account, no login|guest|guest@example.comSunday, March 20, 2011
  12. csv Dialect 9 csv.register_dialect(pipes, delimiter=|) 10 11 reader = csv.reader(sys.stdin, dialect=pipes) 12 for row in reader: 13 print rowSunday, March 20, 2011
  13. csv Dialect 9 csv.register_dialect(pipes, delimiter=|) 10 11 reader = csv.reader(sys.stdin, dialect=pipes) 12 for row in reader: 13 print rowSunday, March 20, 2011
  14. csv Dialect 9 csv.register_dialect(pipes, delimiter=|) 10 11 reader = csv.reader(sys.stdin, dialect=pipes) 12 for row in reader: 13 print rowSunday, March 20, 2011
  15. Database Contents $ ./hidden_stdlib/csv_mkdb.sh $ sqlite3 -noheader test.db select * from users Doug Hellmann|dhellmann|doug.hellmann@gmail.com Guest account, no login|guest|guest@example.com $ sqlite3 -noheader test.db select * from users | python ./hidden_stdlib/csv_pipes.py [Doug Hellmann, dhellmann, doug.hellmann@gmail.com] [Guest account, no login, guest, guest@example.com]Sunday, March 20, 2011
  16. Database Contents $ ./hidden_stdlib/csv_mkdb.sh $ sqlite3 -noheader test.db select * from users Doug Hellmann|dhellmann|doug.hellmann@gmail.com Guest account, no login|guest|guest@example.com $ sqlite3 -noheader test.db select * from users | python ./hidden_stdlib/csv_pipes.py [Doug Hellmann, dhellmann, doug.hellmann@gmail.com] [Guest account, no login, guest, guest@example.com]Sunday, March 20, 2011
  17. Custom Column Types sqlite3Sunday, March 20, 2011
  18. Custom SQLite Column Types 42 # Insert the objects into the database 43 print Inserting: 44 to_save = [ (MyObj(this is a value to save),), 45 (MyObj(42),), 46 ] 47 cursor.executemany( 48 "insert into obj (data) values (?)", 49 to_save) 50 51 # Query the database for the objects just saved 52 print nQuerying: 53 cursor.execute("select id, data from obj") 54 for obj_id, obj in cursor.fetchall(): 55 print Retrieved, obj_id, type(obj), objSunday, March 20, 2011
  19. Custom SQLite Column Types 42 # Insert the objects into the database 43 print Inserting: 44 to_save = [ (MyObj(this is a value to save),), 45 (MyObj(42),), 46 ] 47 cursor.executemany( 48 "insert into obj (data) values (?)", 49 to_save) 50 51 # Query the database for the objects just saved 52 print nQuerying: 53 cursor.execute("select id, data from obj") 54 for obj_id, obj in cursor.fetchall(): 55 print Retrieved, obj_id, type(obj), objSunday, March 20, 2011
  20. Custom SQLite Column Types 42 # Insert the objects into the database 43 print Inserting: 44 to_save = [ (MyObj(this is a value to save),), 45 (MyObj(42),), 46 ] 47 cursor.executemany( 48 "insert into obj (data) values (?)", 49 to_save) 50 51 # Query the database for the objects just saved 52 print nQuerying: 53 cursor.execute("select id, data from obj") 54 for obj_id, obj in cursor.fetchall(): 55 print Retrieved, obj_id, type(obj), objSunday, March 20, 2011
  21. Custom SQLite Column Types 9 class MyObj(object): 10 def __init__(self, arg): 11 self.arg = arg 12 def __str__(self): 13 return MyObj(%r) % self.argSunday, March 20, 2011
  22. Custom SQLite Column Types 17 def adapter_func(obj): 18 """memory -> storage""" 19 print Saving:, obj 20 return pickle.dumps(obj) 21 22 sqlite3.register_adapter(MyObj, adapter_func) 23 24 def converter_func(data): 25 """storage -> memory""" 26 return pickle.loads(data) 27 28 sqlite3.register_converter("MyObj", converter_func)Sunday, March 20, 2011
  23. Custom SQLite Column Types 17 def adapter_func(obj): 18 """memory -> storage""" 19 print Saving:, obj 20 return pickle.dumps(obj) 21 22 sqlite3.register_adapter(MyObj, adapter_func) 23 24 def converter_func(data): 25 """storage -> memory""" 26 return pickle.loads(data) 27 28 sqlite3.register_converter("MyObj", converter_func)Sunday, March 20, 2011
  24. Custom SQLite Column Types 17 def adapter_func(obj): 18 """memory -> storage""" 19 print Saving:, obj 20 return pickle.dumps(obj) 21 22 sqlite3.register_adapter(MyObj, adapter_func) 23 24 def converter_func(data): 25 """storage -> memory""" 26 return pickle.loads(data) 27 28 sqlite3.register_converter("MyObj", converter_func)Sunday, March 20, 2011
  25. Custom SQLite Column Types 17 def adapter_func(obj): 18 """memory -> storage""" 19 print Saving:, obj 20 return pickle.dumps(obj) 21 22 sqlite3.register_adapter(MyObj, adapter_func) 23 24 def converter_func(data): 25 """storage -> memory""" 26 return pickle.loads(data) 27 28 sqlite3.register_converter("MyObj", converter_func)Sunday, March 20, 2011
  26. Custom SQLite Column Types 30 with sqlite3.connect( 31 type_demo.db, 32 detect_types=sqlite3.PARSE_DECLTYPES) as conn: 33 # Create a table with column of type "MyObj" 34 conn.execute(""" 35 create table if not exists obj ( 36 id integer primary key autoincrement not null, 37 data MyObj 38 ) 39 """)Sunday, March 20, 2011
  27. Custom SQLite Column Types 30 with sqlite3.connect( 31 type_demo.db, 32 detect_types=sqlite3.PARSE_DECLTYPES) as conn: 33 # Create a table with column of type "MyObj" 34 conn.execute(""" 35 create table if not exists obj ( 36 id integer primary key autoincrement not null, 37 data MyObj 38 ) 39 """)Sunday, March 20, 2011
  28. Custom SQLite Column Types $ python hidden_stdlib/sqlite3_custom_type.py Inserting: Saving: MyObj(this is a value to save) Saving: MyObj(42) Querying: Retrieved 1 <class __main__.MyObj> MyObj(this is a value to save) Retrieved 2 <class __main__.MyObj> MyObj(42)Sunday, March 20, 2011
  29. Custom SQLite Column Types $ python hidden_stdlib/sqlite3_custom_type.py Inserting: Saving: MyObj(this is a value to save) Saving: MyObj(42) Querying: Retrieved 1 <class __main__.MyObj> MyObj(this is a value to save) Retrieved 2 <class __main__.MyObj> MyObj(42)Sunday, March 20, 2011
  30. Custom SQLite Column Types $ python hidden_stdlib/sqlite3_custom_type.py Inserting: Saving: MyObj(this is a value to save) Saving: MyObj(42) Querying: Retrieved 1 <class __main__.MyObj> MyObj(this is a value to save) Retrieved 2 <class __main__.MyObj> MyObj(42)Sunday, March 20, 2011
  31. Signed Serialized Data hmacSunday, March 20, 2011
  32. hmac 11 message = The message 12 encoded_message = pickle.dumps(message) 13 14 digest_maker = hmac.new(shared-secret-value) 15 digest_maker.update(encoded_message) 16 signature = digest_maker.hexdigest() 17 18 print Outgoing signature:, signature 19 20 # Simulate sending the message 21 buffer = StringIO(%sn%dn%s % (signature, 22 len(encoded_message), 23 encoded_message 24 ))Sunday, March 20, 2011
  33. hmac 11 message = The message 12 encoded_message = pickle.dumps(message) 13 14 digest_maker = hmac.new(shared-secret-value) 15 digest_maker.update(encoded_message) 16 signature = digest_maker.hexdigest() 17 18 print Outgoing signature:, signature 19 20 # Simulate sending the message 21 buffer = StringIO(%sn%dn%s % (signature, 22 len(encoded_message), 23 encoded_message 24 ))Sunday, March 20, 2011
  34. hmac 11 message = The message 12 encoded_message = pickle.dumps(message) 13 14 digest_maker = hmac.new(shared-secret-value) 15 digest_maker.update(encoded_message) 16 signature = digest_maker.hexdigest() 17 18 print Outgoing signature:, signature 19 20 # Simulate sending the message 21 buffer = StringIO(%sn%dn%s % (signature, 22 len(encoded_message), 23 encoded_message 24 ))Sunday, March 20, 2011
  35. hmac 26 # "Receive" the message 27 read_signature = buffer.readline().rstrip() 28 message_len = int(buffer.readline()) 29 read_message = buffer.read(message_len) 30 31 # Check the signature of the incoming data 32 digest_maker = hmac.new(shared-secret-value, 33 read_message) 34 computed_signature = digest_maker.hexdigest() 35 print Computed signature:, signatureSunday, March 20, 2011
  36. hmac 26 # "Receive" the message 27 read_signature = buffer.readline().rstrip() 28 message_len = int(buffer.readline()) 29 read_message = buffer.read(message_len) 30 31 # Check the signature of the incoming data 32 digest_maker = hmac.new(shared-secret-value, 33 read_message) 34 computed_signature = digest_maker.hexdigest() 35 print Computed signature:, signatureSunday, March 20, 2011
  37. hmac 37 if computed_signature == read_signature: 38 print nValid message, processed 39 safe_message = pickle.loads(read_message) 40 print Message:, safe_message 41 else: 42 raise ValueError(Invalid message, discarded)Sunday, March 20, 2011
  38. hmac 37 if computed_signature == read_signature: 38 print nValid message, processed 39 safe_message = pickle.loads(read_message) 40 print Message:, safe_message 41 else: 42 raise ValueError(Invalid message, discarded)Sunday, March 20, 2011
  39. hmac $ python hidden_stdlib/hmac_message_signing.py Outgoing signature: 3498b6fa42a75dc4603da6f4c67505a2 Computed signature: 3498b6fa42a75dc4603da6f4c67505a2 Valid message, processed Message: The messageSunday, March 20, 2011
  40. Serializing Objects jsonSunday, March 20, 2011
  41. json 10 def convert_to_builtin_type(obj): 11 """object->dictionary""" 12 print convert_to_builtin_type(%r) % obj 13 class_name = obj.__class__.__name__ 14 module_name = obj.__module__ 15 digest_maker = hmac.new(PyCon2011, 16 module_name + class_name) 17 signature = digest_maker.hexdigest() 18 d = { __class__:class_name, 19 __module__:module_name, 20 __signature__:signature, 21 } 22 d.update(obj.__dict__) 23 return d 24 25 obj = json_myobj.MyObj(instance value goes here) 26 print json.dumps(obj, default=convert_to_builtin_type)Sunday, March 20, 2011
  42. json 10 def convert_to_builtin_type(obj): 11 """object->dictionary""" 12 print convert_to_builtin_type(%r) % obj 13 class_name = obj.__class__.__name__ 14 module_name = obj.__module__ 15 digest_maker = hmac.new(PyCon2011, 16 module_name + class_name) 17 signature = digest_maker.hexdigest() 18 d = { __class__:class_name, 19 __module__:module_name, 20 __signature__:signature, 21 } 22 d.update(obj.__dict__) 23 return d 24 25 obj = json_myobj.MyObj(instance value goes here) 26 print json.dumps(obj, default=convert_to_builtin_type)Sunday, March 20, 2011
  43. json 10 def convert_to_builtin_type(obj): 11 """object->dictionary""" 12 print convert_to_builtin_type(%r) % obj 13 class_name = obj.__class__.__name__ 14 module_name = obj.__module__ 15 digest_maker = hmac.new(PyCon2011, 16 module_name + class_name) 17 signature = digest_maker.hexdigest() 18 d = { __class__:class_name, 19 __module__:module_name, 20 __signature__:signature, 21 } 22 d.update(obj.__dict__) 23 return d 24 25 obj = json_myobj.MyObj(instance value goes here) 26 print json.dumps(obj, default=convert_to_builtin_type)Sunday, March 20, 2011
  44. json 10 def convert_to_builtin_type(obj): 11 """object->dictionary""" 12 print convert_to_builtin_type(%r) % obj 13 class_name = obj.__class__.__name__ 14 module_name = obj.__module__ 15 digest_maker = hmac.new(PyCon2011, 16 module_name + class_name) 17 signature = digest_maker.hexdigest() 18 d = { __class__:class_name, 19 __module__:module_name, 20 __signature__:signature, 21 } 22 d.update(obj.__dict__) 23 return d 24 25 obj = json_myobj.MyObj(instance value goes here) 26 print json.dumps(obj, default=convert_to_builtin_type)Sunday, March 20, 2011
  45. json $ python hidden_stdlib/json_dump_default.py convert_to_builtin_type(<MyObj(instance value goes here)>) {"s": "instance value goes here", "__module__": "json_myobj", "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", "__class__": "MyObj"}Sunday, March 20, 2011
  46. json $ python hidden_stdlib/json_dump_default.py convert_to_builtin_type(<MyObj(instance value goes here)>) {"s": "instance value goes here", "__module__": "json_myobj", "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", "__class__": "MyObj"}Sunday, March 20, 2011
  47. json $ python hidden_stdlib/json_dump_default.py convert_to_builtin_type(<MyObj(instance value goes here)>) {"s": "instance value goes here", "__module__": "json_myobj", "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", "__class__": "MyObj"}Sunday, March 20, 2011
  48. json $ python hidden_stdlib/json_dump_default.py convert_to_builtin_type(<MyObj(instance value goes here)>) {"s": "instance value goes here", "__module__": "json_myobj", "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", "__class__": "MyObj"}Sunday, March 20, 2011
  49. json $ python hidden_stdlib/json_dump_default.py convert_to_builtin_type(<MyObj(instance value goes here)>) {"s": "instance value goes here", "__module__": "json_myobj", "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", "__class__": "MyObj"}Sunday, March 20, 2011
  50. json 9 def dict_to_object(d): 10 if __class__ not in d: 11 return d 12 13 class_name = d.pop(__class__) 14 module_name = d.pop(__module__) 15 signature = d.pop(__signature__) 16 17 digest_maker = hmac.new(PyCon2011, 18 module_name + class_name) 19 expected_signature = digest_maker.hexdigest() 20 if signature != expected_signature: 21 raise ValueError(Invalid signature)Sunday, March 20, 2011
  51. json 9 def dict_to_object(d): 10 if __class__ not in d: 11 return d 12 13 class_name = d.pop(__class__) 14 module_name = d.pop(__module__) 15 signature = d.pop(__signature__) 16 17 digest_maker = hmac.new(PyCon2011, 18 module_name + class_name) 19 expected_signature = digest_maker.hexdigest() 20 if signature != expected_signature: 21 raise ValueError(Invalid signature)Sunday, March 20, 2011
  52. json 9 def dict_to_object(d): 10 if __class__ not in d: 11 return d 12 13 class_name = d.pop(__class__) 14 module_name = d.pop(__module__) 15 signature = d.pop(__signature__) 16 17 digest_maker = hmac.new(PyCon2011, 18 module_name + class_name) 19 expected_signature = digest_maker.hexdigest() 20 if signature != expected_signature: 21 raise ValueError(Invalid signature)Sunday, March 20, 2011
  53. json 23 print Loading "%s" from "%s" % 24 (class_name, module_name) 25 module = __import__(module_name) 26 class_ = getattr(module, class_name) 27 28 args = dict( (key.encode(ascii), value) 29 for key, value in d.items()) 30 print Instantiating with, args 31 32 inst = class_(**args) 33 return instSunday, March 20, 2011
  54. json 23 print Loading "%s" from "%s" % 24 (class_name, module_name) 25 module = __import__(module_name) 26 class_ = getattr(module, class_name) 27 28 args = dict( (key.encode(ascii), value) 29 for key, value in d.items()) 30 print Instantiating with, args 31 32 inst = class_(**args) 33 return instSunday, March 20, 2011
  55. json 23 print Loading "%s" from "%s" % 24 (class_name, module_name) 25 module = __import__(module_name) 26 class_ = getattr(module, class_name) 27 28 args = dict( (key.encode(ascii), value) 29 for key, value in d.items()) 30 print Instantiating with, args 31 32 inst = class_(**args) 33 return instSunday, March 20, 2011
  56. json 23 print Loading "%s" from "%s" % 24 (class_name, module_name) 25 module = __import__(module_name) 26 class_ = getattr(module, class_name) 27 28 args = dict( (key.encode(ascii), value) 29 for key, value in d.items()) 30 print Instantiating with, args 31 32 inst = class_(**args) 33 return instSunday, March 20, 2011
  57. json 35 for encoded_object in [ 36 37 [{"s": "instance value goes here", 38 "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", 39 "__module__": "json_myobj", "__class__": "MyObj"}] 40 , 41 42 # careful! 43 44 [{"path": "/etc/passwd", 45 "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", 46 "__module__": "os", "__class__": "unlink"}] 47 , 48 ]:Sunday, March 20, 2011
  58. json 35 for encoded_object in [ 36 37 [{"s": "instance value goes here", 38 "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", 39 "__module__": "json_myobj", "__class__": "MyObj"}] 40 , 41 42 # careful! 43 44 [{"path": "/etc/passwd", 45 "__signature__": "426f662f9fe3b3533d9ce7f9dcf8af77", 46 "__module__": "os", "__class__": "unlink"}] 47 , 48 ]:Sunday, March 20, 2011
  59. json 49 try: 50 myobj_instance = json.loads( 51 encoded_object, 52 object_hook=dict_to_object, 53 ) 54 print myobj_instance 55 except Exception as err: 56 print ERROR:, err 57 printSunday, March 20, 2011
  60. json $ python hidden_stdlib/json_load_object_hook.py Loading "MyObj" from "json_myobj" Instantiating with {s: uinstance value goes here} [<MyObj(instance value goes here)>] ERROR: Invalid signatureSunday, March 20, 2011
  61. json $ python hidden_stdlib/json_load_object_hook.py Loading "MyObj" from "json_myobj" Instantiating with {s: uinstance value goes here} [<MyObj(instance value goes here)>] ERROR: Invalid signatureSunday, March 20, 2011
  62. Error Handling sys.excepthookSunday, March 20, 2011
  63. sys.excepthook 6 def main(): 7 # do some work 8 raise RuntimeError(Helpful error message) 9 10 if __name__ == __main__: 11 main() $ python hidden_stdlib/sys_excepthook_no_handler.py Traceback (most recent call last): File ".../hidden_stdlib/sys_excepthook_no_handler.py", line 11, in <module> main() File ".../hidden_stdlib/sys_excepthook_no_handler.py", line 8, in main raise RuntimeError(Helpful error message) RuntimeError: Helpful error messageSunday, March 20, 2011
  64. sys.excepthook 6 import sys 7 8 def main(): 9 try: 10 # do some work 11 raise RuntimeError(Helpful error message) 12 except Exception as err: 13 sys.stderr.write(ERROR: %sn % err) 14 15 if __name__ == __main__: 16 main() $ python hidden_stdlib/sys_excepthook_big_block.py ERROR: Helpful error messageSunday, March 20, 2011
  65. sys.excepthook 6 import sys 7 8 def quiet_errors(exc_type, exc_value, traceback): 9 sys.stderr.write(ERROR: %sn % exc_value) 10 11 sys.excepthook = quiet_errors 12 13 def main(): 14 # do some work 15 raise RuntimeError(Helpful error message) 16 17 if __name__ == __main__: 18 main() $ python hidden_stdlib/sys_excepthook.py ERROR: Helpful error messageSunday, March 20, 2011
  66. sys.excepthook 6 import sys 7 8 def quiet_errors(exc_type, exc_value, traceback): 9 sys.stderr.write(ERROR: %sn % exc_value) 10 11 sys.excepthook = quiet_errors 12 13 def main(): 14 # do some work 15 raise RuntimeError(Helpful error message) 16 17 if __name__ == __main__: 18 main() $ python hidden_stdlib/sys_excepthook.py ERROR: Helpful error messageSunday, March 20, 2011
  67. Communicating logging with UsersSunday, March 20, 2011
  68. logging loggersSunday, March 20, 2011
  69. logging loggersSunday, March 20, 2011
  70. logging 10 # Log verbosely 11 root_logger = logging.getLogger() 12 root_logger.setLevel(logging.DEBUG) 13 14 # Set up console output to stderr 15 console = logging.StreamHandler(sys.stderr) 16 console_format = %(message)s 17 console.setFormatter(logging.Formatter(console_format)) 18 console.setLevel(logging.INFO) # TODO: command line switch 19 root_logger.addHandler(console)Sunday, March 20, 2011
  71. logging 10 # Log verbosely 11 root_logger = logging.getLogger() 12 root_logger.setLevel(logging.DEBUG) 13 14 # Set up console output to stderr 15 console = logging.StreamHandler(sys.stderr) 16 console_format = %(message)s 17 console.setFormatter(logging.Formatter(console_format)) 18 console.setLevel(logging.INFO) # TODO: command line switch 19 root_logger.addHandler(console)Sunday, March 20, 2011
  72. logging 10 # Log verbosely 11 root_logger = logging.getLogger() 12 root_logger.setLevel(logging.DEBUG) 13 14 # Set up console output to stderr 15 console = logging.StreamHandler(sys.stderr) 16 console_format = %(message)s 17 console.setFormatter(logging.Formatter(console_format)) 18 console.setLevel(logging.INFO) # TODO: command line switch 19 root_logger.addHandler(console)Sunday, March 20, 2011
  73. logging 10 # Log verbosely 11 root_logger = logging.getLogger() 12 root_logger.setLevel(logging.DEBUG) 13 14 # Set up console output to stderr 15 console = logging.StreamHandler(sys.stderr) 16 console_format = %(message)s 17 console.setFormatter(logging.Formatter(console_format)) 18 console.setLevel(logging.INFO) # TODO: command line switch 19 root_logger.addHandler(console)Sunday, March 20, 2011
  74. logging 10 # Log verbosely 11 root_logger = logging.getLogger() 12 root_logger.setLevel(logging.DEBUG) 13 14 # Set up console output to stderr 15 console = logging.StreamHandler(sys.stderr) 16 console_format = %(message)s 17 console.setFormatter(logging.Formatter(console_format)) 18 console.setLevel(logging.INFO) # TODO: command line switch 19 root_logger.addHandler(console)Sunday, March 20, 2011
  75. logging 21 # Include debug messages when logging to a file 22 file_handler = logging.handlers.RotatingFileHandler( 23 logging_example.log, # use a full path 24 ) 25 file_format = %(asctime)s %(levelname)6s %(name)s % (message)s 26 file_handler.setFormatter(logging.Formatter(file_format)) 27 file_handler.setLevel(logging.DEBUG) 28 root_logger.addHandler(file_handler)Sunday, March 20, 2011
  76. logging 21 # Include debug messages when logging to a file 22 file_handler = logging.handlers.RotatingFileHandler( 23 logging_example.log, # use a full path 24 ) 25 file_format = %(asctime)s %(levelname)6s %(name)s % (message)s 26 file_handler.setFormatter(logging.Formatter(file_format)) 27 file_handler.setLevel(logging.DEBUG) 28 root_logger.addHandler(file_handler)Sunday, March 20, 2011
  77. logging 21 # Include debug messages when logging to a file 22 file_handler = logging.handlers.RotatingFileHandler( 23 logging_example.log, # use a full path 24 ) 25 file_format = %(asctime)s %(levelname)6s %(name)s % (message)s 26 file_handler.setFormatter(logging.Formatter(file_format)) 27 file_handler.setLevel(logging.DEBUG) 28 root_logger.addHandler(file_handler)Sunday, March 20, 2011
  78. logging 30 # Log sample messages with different levels 31 log = logging.getLogger(__name__) 32 log.info(on the console and in the file) 33 log.debug(only in the file) 34 log.error(simple error message) 35 36 # Replace excepthook with logger 37 def log_exception(exc_type, exc_value, traceback): 38 logging.getLogger(__name__).error(exc_value) 39 sys.excepthook = log_exception 40 41 # Send exceptions to the logger automatically 42 raise RuntimeError(failure message)Sunday, March 20, 2011
  79. logging 30 # Log sample messages with different levels 31 log = logging.getLogger(__name__) 32 log.info(on the console and in the file) 33 log.debug(only in the file) 34 log.error(simple error message) 35 36 # Replace excepthook with logger 37 def log_exception(exc_type, exc_value, traceback): 38 logging.getLogger(__name__).error(exc_value) 39 sys.excepthook = log_exception 40 41 # Send exceptions to the logger automatically 42 raise RuntimeError(failure message)Sunday, March 20, 2011
  80. logging 30 # Log sample messages with different levels 31 log = logging.getLogger(__name__) 32 log.info(on the console and in the file) 33 log.debug(only in the file) 34 log.error(simple error message) 35 36 # Replace excepthook with logger 37 def log_exception(exc_type, exc_value, traceback): 38 logging.getLogger(__name__).error(exc_value) 39 sys.excepthook = log_exception 40 41 # Send exceptions to the logger automatically 42 raise RuntimeError(failure message)Sunday, March 20, 2011
  81. logging $ python hidden_stdlib/logging_example.py on the console and in the file simple error message failure messageSunday, March 20, 2011
  82. logging $ python hidden_stdlib/logging_example.py on the console and in the file simple error message failure message $ cat logging_example.log 2011-02-13 11:28:30,036 INFO __main__ on the console and in the file 2011-02-13 11:28:30,036 DEBUG __main__ only in the file 2011-02-13 11:28:30,036 ERROR __main__ simple error message 2011-02-13 11:28:30,036 ERROR __main__ failure messageSunday, March 20, 2011
  83. logging $ python hidden_stdlib/logging_example.py on the console and in the file simple error message failure message $ cat logging_example.log 2011-02-13 11:28:30,036 INFO __main__ on the console and in the file 2011-02-13 11:28:30,036 DEBUG __main__ only in the file 2011-02-13 11:28:30,036 ERROR __main__ simple error message 2011-02-13 11:28:30,036 ERROR __main__ failure messageSunday, March 20, 2011
  84. logging $ python hidden_stdlib/logging_example.py on the console and in the file simple error message failure message $ cat logging_example.log 2011-02-13 11:28:30,036 INFO __main__ on the console and in the file 2011-02-13 11:28:30,036 DEBUG __main__ only in the file 2011-02-13 11:28:30,036 ERROR __main__ simple error message 2011-02-13 11:28:30,036 ERROR __main__ failure messageSunday, March 20, 2011
  85. logging $ python hidden_stdlib/logging_example.py on the console and in the file simple error message failure message $ cat logging_example.log 2011-02-13 11:28:30,036 INFO __main__ on the console and in the file 2011-02-13 11:28:30,036 DEBUG __main__ only in the file 2011-02-13 11:28:30,036 ERROR __main__ simple error message 2011-02-13 11:28:30,036 ERROR __main__ failure messageSunday, March 20, 2011
  86. logging Traceback HandlerSunday, March 20, 2011
  87. logging 36 # Tracebacks should only go to the file 37 traceback_log = logging.getLogger(traceback) 38 traceback_log.propagate = False 39 traceback_log.setLevel(logging.ERROR) 40 traceback_log.addHandler(file_handler) 41 42 # Replace excepthook with logger 43 def log_exception(exc_type, exc_value, traceback): 44 logging.getLogger(__name__).error(exc_value) 45 logging.getLogger(traceback).error( 46 exc_value, 47 exc_info=(exc_type, exc_value, traceback), 48 ) 49 sys.excepthook = log_exception 50 51 # Send exceptions to the logger automatically 52 raise RuntimeError(failure message)Sunday, March 20, 2011
  88. logging 36 # Tracebacks should only go to the file 37 traceback_log = logging.getLogger(traceback) 38 traceback_log.propagate = False 39 traceback_log.setLevel(logging.ERROR) 40 traceback_log.addHandler(file_handler) 41 42 # Replace excepthook with logger 43 def log_exception(exc_type, exc_value, traceback): 44 logging.getLogger(__name__).error(exc_value) 45 logging.getLogger(traceback).error( 46 exc_value, 47 exc_info=(exc_type, exc_value, traceback), 48 ) 49 sys.excepthook = log_exception 50 51 # Send exceptions to the logger automatically 52 raise RuntimeError(failure message)Sunday, March 20, 2011
  89. logging $ python hidden_stdlib/logging_example_tracebacks.py on the console and in the file simple error message failure message $ cat logging_example.log 2011-02-13 11:33:22,592 INFO __main__ on the console and in the file 2011-02-13 11:33:22,592 DEBUG __main__ only in the file 2011-02-13 11:33:22,592 ERROR __main__ simple error message 2011-02-13 11:33:22,592 ERROR __main__ failure message 2011-02-13 11:33:22,593 ERROR traceback failure message Traceback (most recent call last): File "hidden_stdlib/logging_example_tracebacks.py", line 52, in <module> raise RuntimeError(failure message) RuntimeError: failure messageSunday, March 20, 2011
  90. More Information • http://docs.python.org/ • http://www.doughellmann.com/PyMOTW/ • https://bitbucket.org/dhellmann/hidden_stdlib/ • The Python Standard Library By Example, Doug Hellmann • Python Essential Reference, David BeazleySunday, March 20, 2011
  91. Image Credits 2. http://www.uottawa.ca.libguides.com/content.php?pid=14825&sid=117658 4. http://www.flickr.com/photos/seditiouscanary/1279041211/ 5. http://www.flickr.com/photos/elrentaplats/4902419885/in/photostream/ 6. http://www.flickr.com/photos/elrentaplats/4903005444/in/photostream/ 18. http://etc.usf.edu/clipart/58000/58063/58063_column_types.htm 34. http://www.flickr.com/photos/akeg/3077866744/ 44. http://www.flickr.com/photos/yum9me/2807475045/ 64. http://commons.wikimedia.org/wiki/File:Zeichen_trissturis_error101.png 72. http://www.flickr.com/photos/myklroventine/3545127104/in/ faves-40068198@N04/Sunday, March 20, 2011

×