How I learned to stop worrying & love Python Packaging


Published on

Published in: Technology

How I learned to stop worrying & love Python Packaging

  1. 1. oW Jannis Leidelh jezdez@enn.iod @enn_io
  2. 2. 6How I learned to stop worrying & love Python Packaging
  3. 3. 61) The past of Python packaging2) Common pitfalls and gotchas3) Everyday software development4) The future of Python packaging
  4. 4. Ambiguous terms- Python package A directory with Python files (module) and a file- Release A version of a specific software, result of a development cycle, e.g. “Django 1.3”- Distribution Source or binary form of a release, e.g. zipped tarball or Windows installer
  5. 5. 1The past of Pythonpackaging, a briefhistorical overview
  6. 6. Distutils history“[..] a standard mechanism for building,distributing, and installing Pythonmodules – or, more realistically, multi-module distributions.” Greg Ward- creates distributions- installs to library directory- builds C extensions- processes documentation
  7. 7. Distutils enhancements- PEP 241 (2001) Metadata in PKG-INFO files added to with package distributions- PEP 301 (2002) Package Index and Trove classifiers added and register command- PEP 314 (2003) Metadata 1.1 with license, platform, download URL, dependencies fields
  8. 8. Distutils enhancements- PEP 345 (2005-today) Metadata 1.2, better dependencies- PEP 376 (2009-today) Database of Installed Python packages- PEP 386 (2009-today) Sane version comparison module and better version handling on PyPI
  9. 9. 2Common pitfalls andgotchas of packaging Python software
  10. 10. The standard setup.pyfrom  distutils.core  import  setupsetup(        name=MyApp,        version=0.1.0,        description=My  app  that  does  things,        author=John  Doe,,        license=BSD,        packages=[myapp],        package_data  =  {                myapp:  [                        templates/*.html,                ],        },)
  11. 11. Long descriptionRendered on PyPI with docutilsreStructuredText rendererimport  codecsdef  readme(filename):        file  =,  utf-­‐8)                return  unicode(        #  ...        long_description  =  readme(README.rst),)
  12. 12. package_dataLists all additional, non-Python files of arelease to be installed from a distributionsetup(        #  ...        package_data  =  {                myapp:  [                        static/myapp/*/*,                        templates/myapp/*,                        locale/*/LC_MESSAGES/*                ],    },)
  13. 13. MANIFEST.inTemplate for the “MANIFEST” file thatlists all files to be put in a distributioninclude  README.rstrecursive-­‐include  docs  *.txtrecursive-­‐include  myapp/locale  *.mo  *.porecursive-­‐include  myapp/static  *.css  *.pngrecursive-­‐include  myapp/templates  *.html
  14. 14. Use sane versions, dammit!- 0.18 “Catty And The Major”- .000001- “ cialdev” Don’t do that.- Use PEP 386 for formal versions, e.g. “1.0.1” or “0.5a4” etc- Use release names in any prose documentation and changelogs, e.g. super hero names
  15. 15. 3Using packaging ineveryday’s software development
  16. 16. System package management or not?- Install binary packages globally using your operatings system’s package management, e.g. database adapters, PIL, lxml- Vendorize packages that require modifications or short term patches- Install often updated and your project specific packages locally, e.g. Django, pytz, your own app
  17. 17. virtualenv- Isolated Python environments- Create by topic, e.g. by “branch”, “client”, “milestone”, “staging/prod” etc.- Use virtualenv-wrapper for even more shell helpers- pip included in every virtual environment
  18. 18. pip- Sane installation and uninstallation- freezing and unfreezing of requirements $  server1:~  pip  freeze  >  dev-­‐deps.txt $  server2:~  pip  install  -­‐r  dev-­‐deps.txt- VCS integration for quick and dirty use- PyPI mirror support (PEP 381), e.g.
  19. 19. Simple own package index- HTTP server with directory index feature, e.g. Apache’s DirectoryIndex- PyPI clone Chishop pip  install  -­‐f  http://localhost/dists/ pip  install  -­‐i  http://pypi.corp.local/- Use ~/.pip/pip.conf [install] index-­‐url  =  http://pypi.corp.local/
  20. 20. Local index- Use ad-hoc HTTP server $  python  -­‐m  SimpleHTTPServer  8000- .pydistutils.cfg sdist dir [sdist] dist-­‐dir  =  /var/www/dists- Enables pip download-cache [global] download-­‐cache  =  ~/.pip-­‐downloads
  21. 21. 4 The futureof packagingwith Distutils
  22. 22. What is distutils2?- toolbox with reference implementation for PEP 345, PEP 376, PEP 386- the standalone pysetup tool that installs and uninstalls distributions (e.g. “pysetup install Django”)- metadata available outside of Python file in setup.cfg- already in Python 3 trunk as “packaging”
  23. 23. Other distutils2 features- defined requirements depending on environment Requires-­‐Dist:  bar;  python_version  ==   2.4  or  python_version  ==  2.5 Requires-­‐Python:  >=2.5,<3- Obsoleting other releases Obsoletes-­‐Dist:  OtherProject  (<3.0)
  24. 24. Moving from distutils to distutils2- “pysetup create” for conversion to setup.cfg for forward compatibility- “pysetup generate-setup” for conversion setup.cfg to for backwards compatibility
  25. 25. Example distutils2 setup.cfg[metadata]name  =  myappversion  =  0.1author  =  John  Doeauthor-­‐email  =  john@doe.comsummary  =  My  awesome  appdescription-­‐file  =  README.rsthome-­‐page  =­‐url:  Repository,  =  Development  Status  ::  3  -­‐  AlphaLicense  ::  OSI  Approved  ::  MIT  License[files]packages  =  myappextra_files  =  READMEresources  =      etc/  {confdir}/myapp
  26. 26. Other files that are shipped with distutils2- METADATA- RECORD a list of installed files (CSV formatting)- RESOURCES list of non-python files- INSTALLER name of the installer- REQUESTED exists if distribution wasn’t installed as dependency
  27. 27. 61) Keep your files simple2) Follow the standards3) Use virtualenv/pip4) Look out for distutils2/packaging5) In doubt, read the “Hitchhiker’s Guide to Packaging”
  28. 28. Thanks! oQuestions? W Jannis Leidel h jezdez@enn.ioDon‘t forget to d @enn_iowhile we’rein Amsterdam!