This document discusses Python packaging and improving dependency resolution. It provides an overview of packaging, including creating packages with setup.py and uploading them to a package server. It then discusses challenges with early packaging tools like Distutils and improvements with setuptools, pip, and virtualenv. It also examines how pip handles dependency inconsistencies and the importance of pinning dependencies precisely in requirements.txt. Finally, it recommends hosting your own private package index or proxy to improve reliability.

1. Python Packaging
and how to improve dependency
resolution
Tatiana Al-Chueyr Martins
@tati_alchueyr
PythonDay Pernambuco - 28th September 2013, Recife

2. tati_alchueyr.__doc__
● computer engineer by
UNICAMP
● senior software engineer
at Globo.com
● open source enthusiastic
● pythonist since 2003
#opensource #python #android #arduino

3. Packaging overview
Code Package Package
Server

4. Code
# life.py
def cleanup_house(address):
# ....
def walk_dog(dog_name):
# …
class Man(object):
__doc__ = “”

5. Package
# world.py
from life import Man

6. Package Server
https://pypi.python.org/

7. Packaging overview
Code Package Package
Server

8. Packaging creation
pack upload

9. Packaging creation
pack
# setup.py
(...)
$ python setup.py sdist

10. Packaging creation upload
# ~/.pypirc
[company]
username: Andreia
password: pwd
repository: http://pypi.company.com
$ python setup.py sdist upload -r company

11. Packaging usage
search &
download
use

12. Packaging usage search &
download
use
$ easy_install life
# something.py
from life import Man

13. Package
way to make the code available so developers
can use it

14. Package
setup.py
- contains lots of
metadata
- dependencies
- paths

15. Packages server: Cheese Shop
place where
developers can:
● find packages
● download packages
● upload packages

16. Brief on Python packaging history
● distutils
○ no dependecy management
○ problems between cross-platforms
○ no consistent way to reproduce an installation
○ not all metadata was handled
● setuptools: built on top of distutils
○ introduces easy_install
○ no way to uninstall installed packages
○ provides dependencies management
○ introduced eggs (similar to zip files)
● distribute: fork of setuptools
○ fork of setuptools
● distutils2 (discontinued?)
○ standard versioning (major.minor.micro)
○ setup.cfg: pulls metadata from setup.py file, without needing to run setup.py
○ which operating system requires which dependecy
pysetup: their interations easyinstall and setuptools with disutils- extract stuff from setup.
py

17. Distutils
● Started by Distutils SIG (Greg Ward)
● Added to stand lib in Python 1.6 (2000)
● solves
○ issues a variety of commands through setup.py
(crete tarball, install your project, compiling C
extensions of your python code)
● problems
○ no dependency management
○ problems between OS
○ no consistent way to reproduce an installation
○ not all metadata was handled

18. Brief on Python packaging history
PEP 386: changing the version comparison
modules
PEP 376: database of installed python
distributions
PEP 345: metadata for python software
packages 1.2

19. Chronology of Packaging by Ziade
http://ziade.org/2012/11/17/chronology-of-packaging/

20. Chronology of Packaging by Ziade
http://ziade.org/2012/11/17/chronology-of-packaging/

21. Brief on Python packaging history
old busted new hawtness
setuptools -> distribute
easy_install -> pip
system python -> virtual-env

22. Virtualenv
“virtualenv is a tool to create isolated Python
environments.”
https://pypi.python.org/pypi/virtualenv

23. VirtualenvWrapper
“virtualenvwrapper is a set of extensions to Ian
Bicking's virtualenv tool” -- Doug Hellmann
https://pypi.python.org/pypi/virtualenvwrapper
$ mkvirtualenv <name>
--python=
--no-site-packages=
--system-site-packages=
$ rmvirtualenv
$VIRTUALENVWRAPPER_HOOK_DIR/initialize

24. Pip
A tool for installing and managing Python
packages.
http://www.pip-installer.org/en/latest/index.html
$ pip search numpy
$ pip help
$ pip install flask
$ pip uninstall django
$ pip freeze
--no-deps
--extra-index-url --index-url
--download-cache --proxy --no-install
git / egg / ...
pip install -r requirements.txt

25. Pip
(...)
“This allows users to be in control of
specifying an environment of packages that are
known to work together.”
(...)
http://www.pip-installer.org/en/latest/cookbook.html

26. How Pip deals with dependency
inconsistencies?

27. Pip install -r requirements.txt
B
A
# requirements.txt
B
C
# B/setup.py
A==1.0.0
# C/setup.py
A>=2.0.0 C
what version of A is
installed?
$ pip install -r
requirements.txt

28. Pip install -r requirements.txt
# requirements.txt
B
C
# B/setup.py
A==1.0.0
# C/setup.py
A>=2.0.0
$ pip freeze
A==1.0.0
B==1.0.0
C==1.00.
B
A
C

29. Pip install -r requirements.txt
# requirements.txt
C
B
# B/setup.py
A==1.0.0
# C/setup.py
A>=2.0.0
what happens? error?
$ pip install -r
requirements.txt
B
A
C

30. Pip install -r requirements.txt
# requirements.txt
C
B
# B/setup.py
A==1.0.0
# C/setup.py
A>=2.0.0
$ pip freeze
A==2.0.0
B==1.0.0
C==1.00.
B
A
C

31. Pip install -r requirements.txt
# requirements.txt
C
B
A==1.5.0
# B/setup.py
A==1.0.0
# C/setup.py
A>=2.0.0
what happens? error?
$ pip install -r
requirements.txt
B
A
C

32. Pip install -r requirements.txt
# requirements.txt
C
B
A==1.5.0
# B/setup.py
A==1.0.0
# C/setup.py
A>=2.0.0
$ pip freeze
A==1.5.0
B==1.0.0
C==1.00.
B
A
C

33. Explanation
Considering pip 1.5.4:
● pip doesn’t identify conflicts of interest
between dependency packages
● why?
○ pip solves dependencies analyzing them in a list
○ it only concerns in solving the dependencies of the
package being analyzed at that moment
○ the last package dependencies prevail

34. provided a package at pypi, how do I
know its dependencies?

35. provided a package at pypi, how do I
know its dependencies?
manually looking to them

36. dependencies of a package
if you install a package, you can use:
$ pip show C
To show dependencies, but they don’t contain
versions - only packages names

37. use pipdeptree
$ pip freeze
A==1.0.0
B==1.0.0
C==1.0.0
$ pipdeptree
Warning!!! Possible confusing dependencies found:
* B==1.0.0 -> A [required: ==1.0.0, installed: 1.0.0]
C==1.0.0 -> A [required: >=2.0.0, installed: 1.0.0]
------------------------------------------------------------------------
wsgiref==0.1.2
B==1.0.0
- A [required: ==1.0.0, installed: 1.0.0]
C==1.0.0
- A [required: >=2.0.0, installed: 1.0.0]

38. Does the requirements.txt assure
your environment will be reproduced
always the same?

39. Does the requirements.txt assure
your environment will be reproduced
always the same?
not necessarily

40. requirements.txt
if you want to assert the same behavior in all
installations:
● don’t use >=, <=, >, <
● pin all dependencies (even deps of deps)
● pin exactly (==)

41. some extra notes

42. Have your own pypi / proxy
old versions might be removed from remote
repositories
the repository might be down during a deploy,
and can crash your application

43. Have your own pypi / proxy

44. Have your own pypi / proxy
host a PyPI mirror (bandersnatch, pep381client)
host a PyPI cache (devp)
PyPI server implementations:
● resilient (devpi)
● AWS S3 PyPI server (pypicloud)
● minimalistic PyPI (pypiserver)
● PyPI written in Django (chishop, djangopypi)
Many others..!
At globo.com we have both a PyPI server and a PyPI cache
proxy.

45. dumb ways to manage your
dependencies….

47. 1. understand the tools you use to
manage dependencies
2. keep your dependencies up to date,
but take care with >= / >
3. take care of your cheese-shop

48. use pipdeptree package!

49. thanks!
slideshare: @alchueyr
questions?
Tatiana Al-Chueyr Martins
@tati_alchueyr

50. last note
http://pypi-ranking.info/author