More Related Content
Similar to Python packaging: how did we get here, and where are we going?
Similar to Python packaging: how did we get here, and where are we going? (20)
Recently uploaded
Recently uploaded (20)
Python packaging: how did we get here, and where are we going?
- 1. Python packaging: How did we get here, and where are we going? Thomas Kluyver PyCon UK, Cardiff, 2019
- 3. History 1994: Python 1.0 2019: Python 3.8 2000: Python 2.0 & distutils 2004: Setuptools 2008: pip 2003: PyPI 2007: virtualenv
- 4. History – distutils 1994: Python 1.0 2019: Python 3.8 2000: Python 2.0 & distutils 2004: Setuptools 2008: pip 2003: PyPI 2007: virtualenv setup.py install from distutils.core import setup setup(name = "foo", version = "1.0", packages = [ "package", "package.subpackage" ])
- 5. History - PyPI 1994: Python 1.0 2019: Python 3.8 2000: Python 2.0 & distutils 2004: Setuptools 2008: pip 2003: PyPI 2007: virtualenv
- 6. History – setuptools 1994: Python 1.0 2019: Python 3.8 2000: Python 2.0 & distutils 2004: Setuptools 2008: pip 2003: PyPI 2007: virtualenv ● easy_install ● .egg packages ● Many other features...
- 7. History – pip & virtualenv 1994: Python 1.0 2019: Python 3.8 2000: Python 2.0 & distutils 2004: Setuptools 2008: pip 2003: PyPI 2007: virtualenv ● Uninstall! ● Backtracking on eggs
- 8. Problem setup.py runs on target system ● User tools can’t control installation ● Can’t switch packaging tools ● Slow
- 9. Solutions 1994: Python 1.0 2019: Python 3.8 2000: Python 2.0 & distutils 2004: Setuptools 2008: pip 2013: wheels 2016: manylinux1 wheels 2016: PEP 518 2017: PEP 517 2003: PyPI 2007: virtualenv
- 10. Solutions – wheels 1994: Python 1.0 2019: Python 3.8 2000: Python 2.0 & distutils 2004: Setuptools 2008: pip 2013: wheels 2016: manylinux1 wheels 2016: PEP 518 2017: PEP 517 2003: PyPI 2007: virtualenv ● No script to run ● Pre-built binaries ● For Linux from 2016 ● Flit (2015) ● All you need are wheels
- 13. Solutions – PEP 518 1994: Python 1.0 2019: Python 3.8 2000: Python 2.0 & distutils 2004: Setuptools 2008: pip 2013: wheels 2016: manylinux1 wheels 2016: PEP 518 2017: PEP 517 2003: PyPI 2007: virtualenv ● Packages to install before setup.py ● pyproject.toml [build-system] requires = ["flit"]
- 14. Solutions – PEP 517 1994: Python 1.0 2019: Python 3.8 2000: Python 2.0 & distutils 2004: Setuptools 2008: pip 2013: wheels 2016: manylinux1 wheels 2016: PEP 518 2017: PEP 517 2003: PyPI 2007: virtualenv ● How to make a wheel [build-system] requires = ["flit"] build-backend = "flit.buildapi" ● Took almost 2 years ● >800 messages ● Alternative: PEP 516
- 17. PEP 517 ● Frontend – e.g. pip ● Backend – e.g. setuptools, flit ● 5 hook functions, like: – get_requires_for_build_wheel() – build_wheel()
- 18. Flit ● Declarative metadata ● Pyproject.toml ● Simple packages only [tool.flit.metadata] module = "h5glance" author = "European XFEL Gm author-email = "cas-suppor maintainer = "Thomas Kluyv home-page = "https://githu
- 19. Poetry
- 20. Enscons ● Scons build system ● Good for compiled code extension = env.SharedLibrary( target=ext_filename, source=[ "crypt_blowfish-1.2/crypt "crypt_blowfish-1.2/crypt "crypt_blowfish-1.2/wrapp "cryptacular/bcrypt/_bcry ], LIBPREFIX="", SHLIBSUFFIX=SHLIBSUFFIX, CPPPATH=["crypt_blowfish-1.2" CPPFLAGS=["-D__SKIP_GNU"],
- 21. Make your own backend ● 4 key functions – build_sdist / _wheel – get_requires_for_build_sdist / _wheel ● Can extend another backend, e.g. flit, setuptools
- 22. Make your own frontend Use the pep517 library! Why? ● Repackaging, e.g. rpm, installers, Spack ● Build packages in docker containers ● Reproducible builds?
- 23. Thank-you Thomas Kluyver thomas@kluyver.me.uk – takluyver (GitHub/GitLab) Thanks to: Nathaniel Smith, Robert Collins, Nick Coghlan, Daniel Holth, Donald Stufft, Paul Moore & many other people