Discussing Python packaging formats and tools.

1. Rolling with the Times
Using Wheels, pbr, and Twine
for Distributing and Installing Software
Doug Hellmann
PyATL Feb 2015

2. Dist Formats
• sdist
• source only
• requires compiler to install C extensions
• egg
• binary importable format
• defacto standard
• wheel
• binary installable format
• follows PyPA team standards

3. Wheel File Names
{distribution}-{version}(-{build tag})?-
{python tag}-{abi tag}-{platform tag}.whl
• distribution – software name
• version-buildtag – version string for distribution
• python tag – version of python
• abitag – C extension application binary interface
• platform tag – CPU architecture

4. Building sdists
$ python setup.py sdist
…
$ tar ztf dist/entry_point_inspector-0.1.tar.gz
entry_point_inspector-0.1/
entry_point_inspector-0.1/entry_point_inspector/
entry_point_inspector-0.1/entry_point_inspector/__init__.py
entry_point_inspector-0.1/entry_point_inspector/app.py
entry_point_inspector-0.1/entry_point_inspector/ep.py
entry_point_inspector-0.1/entry_point_inspector/group.py
entry_point_inspector-0.1/entry_point_inspector.egg-info/
entry_point_inspector-0.1/entry_point_inspector.egg-info/dependency_links.txt
entry_point_inspector-0.1/entry_point_inspector.egg-info/entry_points.txt
entry_point_inspector-0.1/entry_point_inspector.egg-info/namespace_packages.txt
entry_point_inspector-0.1/entry_point_inspector.egg-info/not-zip-safe
entry_point_inspector-0.1/entry_point_inspector.egg-info/PKG-INFO
entry_point_inspector-0.1/entry_point_inspector.egg-info/requires.txt
entry_point_inspector-0.1/entry_point_inspector.egg-info/SOURCES.txt
entry_point_inspector-0.1/entry_point_inspector.egg-info/top_level.txt
entry_point_inspector-0.1/PKG-INFO
entry_point_inspector-0.1/README.rst
entry_point_inspector-0.1/setup.cfg
entry_point_inspector-0.1/setup.py

5. Building eggs
$ python setup.py bdist_egg
…
$ unzip -l dist/entry_point_inspector-0.1-py2.7.egg
Archive: dist/entry_point_inspector-0.1-py2.7.egg
Length Date Time Name
-------- ---- ---- ----
1 01-02-15 11:50 EGG-INFO/dependency_links.txt
224 01-02-15 11:50 EGG-INFO/entry_points.txt
1 01-02-15 11:50 EGG-INFO/namespace_packages.txt
1 07-26-13 15:47 EGG-INFO/not-zip-safe
5132 01-02-15 11:50 EGG-INFO/PKG-INFO
6 01-02-15 11:50 EGG-INFO/requires.txt
511 01-02-15 11:50 EGG-INFO/SOURCES.txt
22 01-02-15 11:50 EGG-INFO/top_level.txt
0 07-26-13 15:49 entry_point_inspector/__init__.py
154 01-02-15 11:50 entry_point_inspector/__init__.pyc
659 07-26-13 17:12 entry_point_inspector/app.py
1346 01-02-15 11:50 entry_point_inspector/app.pyc
2148 07-26-13 17:13 entry_point_inspector/ep.py
2264 01-02-15 11:50 entry_point_inspector/ep.pyc
1527 07-26-13 16:59 entry_point_inspector/group.py
2503 01-02-15 11:50 entry_point_inspector/group.pyc
-------- -------
16499 16 files

6. Building Wheels
$ python setup.py bdist_wheel
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2
[cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command ‘bdist_wheel'

7. Building Wheels
$ python setup.py bdist_wheel
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2
[cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command ‘bdist_wheel'
$ pip install wheel
Collecting wheel
Using cached wheel-0.24.0-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.24.0

8. Building Wheels
$ python setup.py bdist_wheel
…
$ unzip -l dist/entry_point_inspector-0.1-py2-none-any.whl
Archive: dist/entry_point_inspector-0.1-py2-none-any.whl
Length Date Time Name
-------- ---- ---- ----
0 07-26-13 15:49 entry_point_inspector/__init__.py
659 07-26-13 17:12 entry_point_inspector/app.py
2148 07-26-13 17:13 entry_point_inspector/ep.py
1527 07-26-13 16:59 entry_point_inspector/group.py
3761 01-02-15 11:59 entry_point_inspector-0.1.dist-info/DESCRIPTION.rst
224 01-02-15 11:59 entry_point_inspector-0.1.dist-info/entry_points.txt
1265 01-02-15 11:59 entry_point_inspector-0.1.dist-info/metadata.json
1 01-02-15 11:59 entry_point_inspector-0.1.dist-info/
namespace_packages.txt
22 01-02-15 11:59 entry_point_inspector-0.1.dist-info/top_level.txt
92 01-02-15 11:59 entry_point_inspector-0.1.dist-info/WHEEL
4518 01-02-15 11:59 entry_point_inspector-0.1.dist-info/METADATA
1134 01-02-15 11:59 entry_point_inspector-0.1.dist-info/RECORD
-------- -------
15351 12 files

9. Wheel File Contents
• .py source files
• .so/.dll library files
• dist-info standard metadata
• no setup.py

10. sdist Installation
$ mktmpenv
$ pip install --no-cache-dir entry_point_inspector
Collecting entry-point-inspector
Downloading entry_point_inspector-0.1.tar.gz
Collecting cliff (from entry-point-inspector)
Downloading cliff-1.9.0-py2-none-any.whl
Collecting cmd2>=0.6.7 (from cliff->entry-point-inspector)
Downloading cmd2-0.6.8.tar.gz
Collecting stevedore>=1.1.0 (from cliff->entry-point-inspector)
Downloading stevedore-1.1.0-py2.py3-none-any.whl
Collecting pyparsing>=2.0.1 (from cliff->entry-point-inspector)
Downloading pyparsing-2.0.3-py2.py3-none-any.whl
Collecting pbr!=0.7,<1.0,>=0.6 (from cliff->entry-point-inspector)
Downloading pbr-0.10.7-py2.py3-none-any.whl (70kB)
100% |################################| 73kB 1.0MB/s
Collecting PrettyTable<0.8,>=0.7 (from cliff->entry-point-inspector)
Downloading prettytable-0.7.2.tar.bz2
Requirement already satisfied (use --upgrade to upgrade): pip in ./lib/python2.7/site-packages (from pbr!
=0.7,<1.0,>=0.6->cliff->entry-point-inspector)
Installing collected packages: PrettyTable, pbr, pyparsing, stevedore, argparse, six, cmd2, cliff, entry-point-
inspector
Running setup.py install for PrettyTable
Running setup.py install for cmd2
Running setup.py install for entry-point-inspector
Installing epi script to /Users/dhellmann/Envs/tmp-14f1bec315f89c0c/bin
Successfully installed PrettyTable-0.7.2 argparse-1.3.0 cliff-1.9.0 cmd2-0.6.8 entry-point-inspector-0.1
pbr-0.10.7 pyparsing-2.0.3 six-1.9.0 stevedore-1.1.0

11. Speeding Up Installation
$ cat ~/.pip/pip.conf
[global]
wheel-dir = /Users/dhellmann/.pip/wheelhouse
find-links = /Users/dhellmann/.pip/wheelhouse

12. Building Wheels of Other
Packages
$ pip wheel cmd2
Collecting cmd2
Using cached cmd2-0.6.8.tar.gz
Collecting pyparsing>=2.0.1 (from cmd2)
Using cached pyparsing-2.0.3-py2.py3-none-any.whl
Saved /Users/dhellmann/.pip/wheelhouse/pyparsing-2.0.3-
py2.py3-none-any.whl
Skipping pyparsing, due to already being wheel.
Building wheels for collected packages: cmd2
Running setup.py bdist_wheel for cmd2
Destination directory: /Users/dhellmann/.pip/wheelhouse
Successfully built cmd2

13. Wheel Installation
$ mktmpenv
$ pip install entry_point_inspector
Collecting entry-point-inspector
Collecting cliff (from entry-point-inspector)
Collecting cmd2>=0.6.7 (from cliff->entry-point-inspector)
Collecting six>=1.7.0 (from cliff->entry-point-inspector)
Collecting argparse (from cliff->entry-point-inspector)
Collecting stevedore>=1.1.0 (from cliff->entry-point-inspector)
Collecting pyparsing>=2.0.1 (from cliff->entry-point-inspector)
Collecting pbr!=0.7,<1.0,>=0.6 (from cliff->entry-point-inspector)
Collecting PrettyTable<0.8,>=0.7 (from cliff->entry-point-inspector)
Requirement already satisfied (use --upgrade to upgrade): pip in ./lib/python2.7/site-packages (from
pbr!=0.7,<1.0,>=0.6->cliff->entry-point-inspector)
Installing collected packages: PrettyTable, pbr, pyparsing, stevedore, argparse, six, cmd2, cliff,
entry-point-inspector
Successfully installed PrettyTable-0.7.2 argparse-1.3.0 cliff-1.9.0 cmd2-0.6.8 entry-point-
inspector-0.1 pbr-0.10.7 pyparsing-2.0.3 six-1.9.0 stevedore-1.1.0

14. pbr – Python Build
Reasonableness
• Automatically gather meta-data
• Static meta-data as data, not code
• Automatically generate release files

15. pbr – Gathered Meta-data
• git ls → SOURCES.txt
• requirements.txt → install_requires
• README.rst → long_description
• git tag → version

16. pbr – Static Meta-data
• Name
• Trove Classifiers
• Home Page
• Python Packages
• Entry Points

17. pbr – Automatically
Generate Release Files
• AUTHORS
• ChangeLog

18. pbr – setup.py
#!/usr/bin/env python
import setuptools
setuptools.setup(
setup_requires=['pbr'],
pbr=True)

19. pbr – setup.cfg
[metadata]
name = oslo.config
author = OpenStack
author-email = openstack-
dev@lists.openstack.org
summary = Oslo Configuration API
description-file =
README.rst
home-page = https://launchpad.net/oslo
classifier =
Development Status :: 4 - Beta
Environment :: OpenStack
Intended Audience :: Developers
Intended Audience :: Information
Technology
License :: OSI Approved :: Apache
Software License
Operating System :: OS Independent
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.3
[files]
packages =
oslo_config

20. twine – Upload dists to PyPI
• Upload Existing Packages
• Supports sdist, egg, & wheel
• GPG Signature Files

21. twine
$ twine -h
usage: twine [-h] [--version] {upload}
positional arguments:
{upload}
optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit

22. twine – Uploading Signed
dists
python setup.py sdist
python setup.py bdist_wheel
gpg --detach-sign -a $sdistfile
gpg --detach-sign -a $wheelfile
twine upload $sdistfile $sdistsignfile
$wheelfile $wheelsignfile

23. References
• https://www.python.org/dev/peps/pep-0427/
• http://lucumr.pocoo.org/2014/1/27/python-on-
wheels/
• http://pythonwheels.com
• https://www.berrange.com/posts/2014/11/14/faster-
rebuilds-for-python-virtualenv-trees/