This document discusses six Python packages that are useful to know: 1. First - A utility for selecting the first successful result from a sequence of functions. 2. Parse - A library for parsing Python format strings and extracting values. 3. Filecmp - A module for comparing files and directories. 4. Bitrot - A tool for detecting silent data corruption in files. 5. Docopt - A tool for generating command-line interfaces from a docstring. 6. Six - A library for writing code that is compatible with both Python 2 and Python 3.

1. +
FIVE
PYTHON
PACKAGES YOU
NEED TO KNOW

2. +
ambv #python-dev
Łukasz Langa @llanga Twitter
lukasz@langa.pl

3. +
1
pip install first

4. +
pip install first
>>> quiet = False
>>> if not quiet:
... print("O HAI PYCON!")
...
O HAI PYCON!

5. +
pip install first
>>> error_count = 2
>>> if error_count:
... sys.exit(1)
...
ambv@host $

6. +
pip install first
>>> leftovers = ['foo', 'bar']
>>> if leftovers:
... print('{} things left.'
...
''.format(len(leftovers)))
...
2 things left.

7. +
pip install first
>>> leftovers = [0, None]
>>> if leftovers:
... print('{} things left.'
...
''.format(len(leftovers)))
...
2 things left.

8. +
pip install first
>>> leftovers = [0, None]
>>> if any(leftovers):
... print('{} things left.'
...
''.format(len(leftovers)))
...
>>>

9. +
pip install first
>>> any([0, 1, 2])
True

10. +
pip install first
>>> any([0, 1, 2])
True
>>> first([0, 1, 2])
1

11. +
pip install first
s = 'abc'
m = re1.match(s)
if m:
print('re1', m.group(1))
else:
m = re2.match(s)
if m:
print('re2', m.group(1))
else:
m = re3.match(s)
if m:
print('re3', m.group(1))
else:
print('no match!')

12. +
pip install first
s = 'abc'
m = first(r.match(s)
for r in [re1, re2, re3])
if not m:
print('no match!’)
elif m.re is re1:
print('re1', m.group(1))
elif m.re is re2:
print('re2', m.group(1))
elif m.re is re3:
print('re2', m.group(1))

13. +
{}
2
pip install parse

14. +
pip install parse
Old and busted
>>> old_fmt = "Number %f and something else: %s"
>>> old_fmt % (1.23456, 'u0141ukasz')
'Number 1.234560 and something else: Łukasz'

15. +
pip install parse
New hotness
>>> fmt = "Number {:f} and something else: {}"
>>> fmt.format(1.23456, 'u0141ukasz')
'Number 1.234560 and something else: Łukasz'

16. +
pip install parse
New hotness
>>> fmt = "Number {:f} and something else: {}"
>>> fmt.format(1.23456, 'u0141ukasz')
'Number 1.234560 and something else: Łukasz'
>>> concrete_string = _
>>> parse(fmt, concrete_string)
<Result (1.23456, 'Łukasz') {}>
>>> r = _
>>> r[0]
1.23456
>>> r[1]
'Łukasz'

17. +
3
filecmp

18. +
filecmp
$ tree /tmp/dir1
.
├── dirdir1
│ ├── fileA
│ └── fileB
├── file1
├── file2
└── file3

19. +
filecmp
>>> import filecmp
>>> filecmp.cmp('/tmp/dir1/file1',
... '/tmp/dir1/file2')
False
>>> filecmp.cmp('/tmp/dir1/file1',
... '/tmp/dir1/file3')
True

20. +
filecmp
>>> import filecmp
>>> filecmp.cmp('/tmp/dir1/file1',
... '/tmp/dir1/file2')
False
>>> filecmp.cmp('/tmp/dir1/file1',
... '/tmp/dir1/file3')
True

21. +
filecmp
>>> import filecmp
>>> filecmp.cmp('/tmp/dir1/file1',
... '/tmp/dir1/file2')
False
>>> filecmp.cmp('/tmp/dir1/file1',
... '/tmp/dir1/file3')
True

22. +
filecmp
>>> result = filecmp.dircmp('/tmp/dir1',
'/tmp/dir2')
>>> result.report()
diff /tmp/dir1 /tmp/dir2
Identical files : ['file1', 'file3']
Differing files : ['file2']
Common subdirectories : ['dirdir1']
>>> result.report_full_closure()
diff /tmp/dir1 /tmp/dir2
Identical files : ['file1', 'file3']
Differing files : ['file2']
Common subdirectories : ['dirdir1']
diff /tmp/dir1/dirdir1 /tmp/dir2/dirdir1
Only in /tmp/dir1/dirdir1 : ['fileB']
Identical files : ['fileA']

23. +
filecmp
>>> result = filecmp.dircmp('/tmp/dir1',
'/tmp/dir3')
>>> result.report()
diff /tmp/dir1 /tmp/dir3
Identical files : ['file1', 'file2', 'file3']
Common subdirectories : ['dirdir1']
>>> result.report_full_closure()
diff /tmp/dir1 /tmp/dir3
Identical files : ['file1', 'file2', 'file3']
Common subdirectories : ['dirdir1']
diff /tmp/dir1/dirdir1 /tmp/dir3/dirdir1

24. +
4
pip install bitrot

25. +
pip install bitrot
$ cd /tmp/dir1
$ bitrot
Finished. 0.00 MiB of data read. 0 errors found.
5 entries in the database, 5 new, 0 updated,
0 renamed, 0 missing.

26. +
pip install bitrot
$ sqlite3 .bitrot.db
sqlite> select * from sqlite_master where
name='bitrot';
table|bitrot|bitrot|2|CREATE TABLE bitrot
(path TEXT PRIMARY KEY, mtime INTEGER, hash TEXT,
timestamp TEXT)
sqlite> select * from bitrot limit 1;
./dirdir1/fileA|1363385094|5183b6e4557a39428eb00d4a1
4437e45c448ccc6|
2013-03-15 22:29:46

27. +
pip install bitrot
$ bitrot
Finished. 0.00 MiB of data read. 0 errors found.
5 entries in the database, 0 new, 0 updated,
0 renamed, 0 missing.

28. +
pip install bitrot
$ mv file2 file4
$ rm file3
$ echo "Another line." >>file1
$ bitrot
Finished. 0.00 MiB of data read. 0 errors found.
4 entries in the database, 0 new, 1 updated,
1 renamed, 1 missing.

29. +
pip install bitrot
$ bitrot
error: SHA1 mismatch for ./file4:
expected cd3be7e3d60016ce01a835c562e2748cf2cbe596,
got 3875db357b18eb06fae627d1dc0320efa89c2006.
Original info from 2013-03-15 23:29:46.
Finished. 0.00 MiB of data read. 1 error found.
4 entries in the database, 0 new, 0 updated,
0 renamed, 0 missing.

30. +
5
pip install docopt

31. +
pip install docopt
$ rename --help
usage: rename [-h] [-c] [-I] [-l] [-q] [-U] [-v EXCEPT_REGEX] [-t]
[--index-first INDEX_FIRST] [--index-step INDEX_STEP]
[--index-digits INDEX_DIGITS] [--index-pad-with INDEX_PAD_WITH]
[-s] [--selftest [use_directory]]
regex target
positional arguments:
regex regular expression to match files with
target target pattern using references to groups in the
regular expression
optional arguments:
-h, --help show this help message and exit
-c, --copy copy files instead of renaming
-I, --case-insensitive
treat the regular expression as case-insensitive
-l, --lower translate all letters to lower-case
-q, --quiet don't print anything, just return status codes
-U, --upper translate all letters to upper-case
-v EXCEPT_REGEX, --except EXCEPT_REGEX
exclude files matching the following regular
expression
-t, --test test only, don't actually rename anything
-s, --simple invokes the simple mode. For more help on its
positional arguments: rename -s –help
--selftest [use_directory]
run internal unit tests

32. +
pip install docopt
stparser = argparse.ArgumentParser(prog='rename', add_help=False)
classic = argparse.ArgumentParser(prog='rename')
simple = argparse.ArgumentParser(prog='rename')
invocator = Proxy(classic, stparser)
simple.add_argument('-s', '--simple', action='store_true', help='invokes '
'the simple mode', required=True)
common = Proxy(classic, simple)
common.add_argument('-c', '--copy', action='store_true',
help='copy files instead of renaming')
common.add_argument('-I', '--case-insensitive', action='store_true',
help='treat the regular expression as case-insensitive')
common.add_argument('-l', '--lower', action='store_const', dest='xform',
const='lower', help='translate all letters to lower-case')
common.add_argument('-q', '--quiet', action='store_true',
help='don't print anything, just return status codes')
common.add_argument('-U', '--upper', action='store_const', dest='xform',
const='upper', help='translate all letters to upper-case')
common.add_argument('-v', '--except', dest='except_regex', action='store',
default="", help='exclude files matching the following '
'regular expression')
common.add_argument('-t', '--test', action='store_true', help='test only, '
'don't actually rename anything')
group = classic.add_argument_group('Configuration for the special '
'(index) reference')
group.add_argument('--index-first', default=1, help='specifies '
'what number will the first (index) substitution contain. Default: 1')
group.add_argument('--index-step', default=1, help='specifies '
'what number will be added with each step to the first value. Negative '
'numbers allowed. Default: 1')
group.add_argument('--index-digits', default='auto',
help='specifies how many digits will be used in each (index) '
'substitution. If a number has fewer digits, they will be prefixed by '
'leading zeroes (or another character, see --index-pad-with). Default: '
'auto (e.g. path enough digits so that each number uses the same amount'
' of characters)')
group.add_argument('--index-pad-with', default='0',
help='specifies what character will be used for padding. Default: "0"')
invocator.add_argument('-s', '--simple', action='store_true', help='invokes '
'the simple mode. For more help on its positional arguments: '
'rename -s --help')
invocator.add_argument('--selftest', nargs='?', const=True,
metavar='use_directory', help='run internal unit tests')
classic.add_argument('regex', help='regular expression to match '
'files with')
classic.add_argument('target', help='target pattern using '
'references to groups in the regular expression')
simple.add_argument('substring_from', help='simple (raw) substring that '
'should be found within the filename')
simple.add_argument('substring_to', help='the replacement string')
simple.add_argument('regex', help='regular expression to match '
'files with')
args = stparser.parse_known_args()
if args[0].selftest:
selftest(args[0].selftest)
elif args[0].simple:
args = simple.parse_args()
args.regex = "".join(args.regex)
args.substring_from = "".join(args.substring_from)
args.substring_to = "".join(args.substring_to)
sys.exit(Renamer(**args.__dict__).rename_simple(**args.__dict__))
else:
args = classic.parse_args()
args.regex = "".join(args.regex)
args.except_regex = "".join(args.except_regex)
args.target = "".join(args.target)

33. +
pip install docopt
$ httproxy --help
Tiny HTTP Proxy.
This module implements GET, HEAD, POST, PUT, DELETE and CONNECT
methods on BaseHTTPServer.
Usage:
httproxy [options]
httproxy [options] <allowed-client> ...
Options:
-h, --help Show this screen.
--version Show version and exit.
-H, --host HOST Host to bind to [default: 127.0.0.1].
-p, --port PORT Port to bind to [default: 8000].
-l, --logfile PATH Path to the logfile [default: STDOUT].
-i, --pidfile PIDFILE Path to the pidfile [default: httproxy.pid].
-d, --daemon Daemonize (run in the background). The
default logfile path is httproxy.log in
this case.
-c, --configfile CONFIGFILE Path to a configuration file.
-v, --verbose Log headers.

34. +
pip install docopt
"""Tiny HTTP Proxy.
This module implements GET, HEAD, POST, PUT, DELETE and CONNECT
methods on BaseHTTPServer.
Usage:
httproxy [options]
httproxy [options] <allowed-client> ...
Options:
-h, --help Show this screen.
--version Show version and exit.
-H, --host HOST Host to bind to [default: 127.0.0.1].
-p, --port PORT Port to bind to [default: 8000].
-l, --logfile PATH Path to the logfile [default: STDOUT].
-i, --pidfile PIDFILE Path to the pidfile [default: httproxy.pid].
-d, --daemon Daemonize (run in the background). The
default logfile path is httproxy.log in
this case.
-c, --configfile CONFIGFILE Path to a configuration file.
-v, --verbose Log headers.
"""

35. +
pip install docopt
cmdline_args = docopt(
__doc__,
version=__version__,
)

36. +
6
pip install six

37. +
pip install six
Have a Python 2.x project?
1. Write those tests, seriously.
2. Target 2.7 with __futures__:
 division
 print_function
 unicode_literals
3. Run tests on 3.3. Use six to fix leftovers.
4. There is no step 4! *

38. +
pip install six
* Well, to have all bases covered, read:
http://docs.python.org/3/
howto/pyporting.html

39. +
pip install six
Have a Python 2.x project?
1. Tests.
2. 2.7 + three futures.
3. Run tests on both 2.7 and 3.3.
Fix whatever doesn’t work using six.

40. +
pip install six
six.text_type / six.binary_type

41. +
pip install six
six.text_type / six.binary_type
six.iterkeys / six.iteritems / six.itervalues

42. +
pip install six
six.text_type / six.binary_type
six.iterkeys / six.iteritems / six.itervalues
six.next

43. +
pip install six
six.text_type / six.binary_type
six.iterkeys / six.iteritems / six.itervalues
six.next
six.with_metaclass

44. +
pip install six
six.text_type / six.binary_type
six.iterkeys / six.iteritems / six.itervalues
six.next
six.with_metaclass
from six.moves import xrange

45. +
pip install six
six.text_type / six.binary_type
six.iterkeys / six.iteritems / six.itervalues
six.next
six.with_metaclass
from six.moves import xrange
if six.PY3:

46. +
7
pip install tox

47. +
pip install tox
tox.ini
[tox]
envlist = py27,py32,py33
[testenv]
changedir = test
commands =
{envbindir}/python run_tests.py
[testenv:py27]
basepython = python2.7
deps =
configparser

48. +
pip install tox
$ tox
...
_________________ summary _________________
py26: commands succeeded
py27: commands succeeded
py32: commands succeeded
py33: commands succeeded
congratulations :)

49. +
BONUS
Travis-CI

50. +
Travis-CI
.travis.yml
language: python
before_install:
- pip install tox
script: tox

51. +
Travis-CI

52. +
ambv #python-dev
Łukasz Langa @llanga Twitter
lukasz@langa.pl