An overview of Python packaging history, with details of what PEP 517 is, some of the new tools using it, and how you might write a PEP 517 backends and frontends. See also notes taken by an attendee: https://twitter.com/drakekin/status/1173195932151746561

1. Python packaging:
How did we get here,
and where are we going?
Thomas Kluyver
PyCon UK, Cardiff, 2019

3. History
1994: Python 1.0
2019: Python 3.8
2000: Python 2.0 & distutils
2004: Setuptools
2008: pip
2003: PyPI
2007: virtualenv

4. History – distutils
1994: Python 1.0
2019: Python 3.8
2000: Python 2.0 & distutils
2004: Setuptools
2008: pip
2003: PyPI
2007: virtualenv
setup.py install
from distutils.core import setup
setup(name = "foo",
version = "1.0",
packages = [
"package",
"package.subpackage"
])

5. History - PyPI
1994: Python 1.0
2019: Python 3.8
2000: Python 2.0 & distutils
2004: Setuptools
2008: pip
2003: PyPI
2007: virtualenv

6. History – setuptools
1994: Python 1.0
2019: Python 3.8
2000: Python 2.0 & distutils
2004: Setuptools
2008: pip
2003: PyPI
2007: virtualenv
● easy_install
●
.egg packages
●
Many other features...

7. History – pip & virtualenv
1994: Python 1.0
2019: Python 3.8
2000: Python 2.0 & distutils
2004: Setuptools
2008: pip
2003: PyPI
2007: virtualenv
●
Uninstall!
●
Backtracking on eggs

8. Problem
setup.py runs on target system
●
User tools can’t control installation
●
Can’t switch packaging tools
●
Slow

9. Solutions
1994: Python 1.0
2019: Python 3.8
2000: Python 2.0 & distutils
2004: Setuptools
2008: pip 2013: wheels
2016: manylinux1 wheels
2016: PEP 518
2017: PEP 517
2003: PyPI
2007: virtualenv

10. Solutions – wheels
1994: Python 1.0
2019: Python 3.8
2000: Python 2.0 & distutils
2004: Setuptools
2008: pip 2013: wheels
2016: manylinux1 wheels
2016: PEP 518
2017: PEP 517
2003: PyPI
2007: virtualenv
●
No script to run
●
Pre-built binaries
●
For Linux from 2016
●
Flit (2015)
●
All you need are wheels

11. Source
folder
setup.py install
Installed
sdist
(.tar.gz)
setup.py
sdist Before wheels

12. Source
folder
setup.py install
Installed
sdist
(.tar.gz)
wheel
(.whl)
setup.py
bdist_wheel
Before PEP 517

13. Solutions – PEP 518
1994: Python 1.0
2019: Python 3.8
2000: Python 2.0 & distutils
2004: Setuptools
2008: pip 2013: wheels
2016: manylinux1 wheels
2016: PEP 518
2017: PEP 517
2003: PyPI
2007: virtualenv
● Packages to install before setup.py
● pyproject.toml
[build-system]
requires = ["flit"]

14. Solutions – PEP 517
1994: Python 1.0
2019: Python 3.8
2000: Python 2.0 & distutils
2004: Setuptools
2008: pip 2013: wheels
2016: manylinux1 wheels
2016: PEP 518
2017: PEP 517
2003: PyPI
2007: virtualenv
●
How to make a wheel
[build-system]
requires = ["flit"]
build-backend = "flit.buildapi"
●
Took almost 2 years
●
>800 messages
●
Alternative: PEP 516

15. Source
folder
setup.py install
Installed
sdist
(.tar.gz)
wheel
(.whl)
setup.py
bdist_wheel
Today
PEP 517

16. Source
folder
Installed
sdist
(.tar.gz)
wheel
(.whl)
Future
PEP 517

17. PEP 517
●
Frontend – e.g. pip
●
Backend – e.g. setuptools, flit
●
5 hook functions, like:
– get_requires_for_build_wheel()
– build_wheel()

18. Flit
●
Declarative metadata
●
Pyproject.toml
●
Simple packages only
[tool.flit.metadata]
module = "h5glance"
author = "European XFEL Gm
author-email = "cas-suppor
maintainer = "Thomas Kluyv
home-page = "https://githu

19. Poetry

20. Enscons
●
Scons build
system
●
Good for
compiled
code
extension = env.SharedLibrary(
target=ext_filename,
source=[
"crypt_blowfish-1.2/crypt
"crypt_blowfish-1.2/crypt
"crypt_blowfish-1.2/wrapp
"cryptacular/bcrypt/_bcry
],
LIBPREFIX="",
SHLIBSUFFIX=SHLIBSUFFIX,
CPPPATH=["crypt_blowfish-1.2"
CPPFLAGS=["-D__SKIP_GNU"],

21. Make your own backend
●
4 key functions
– build_sdist / _wheel
– get_requires_for_build_sdist / _wheel
●
Can extend another backend, e.g. flit,
setuptools

22. Make your own frontend
Use the pep517 library!
Why?
●
Repackaging, e.g. rpm, installers, Spack
●
Build packages in docker containers
●
Reproducible builds?

23. Thank-you
Thomas Kluyver
thomas@kluyver.me.uk – takluyver (GitHub/GitLab)
Thanks to:
Nathaniel Smith, Robert Collins, Nick Coghlan, Daniel
Holth, Donald Stufft, Paul Moore
& many other people