This document provides a history of Python packaging and discusses lessons learned. It covers: 1. The evolution of Python packaging standards and tools from 1991 to present, including changes to metadata schemas, build tools, package publishing/indexing, installers, and support for multiple platforms/versions. 2. Challenges that existed before and after the community developed packaging solutions, such as companies initially developing their own internal tools and later adopting community standards. 3. How the community's open solutions grew faster than any individual company could achieve, and how problems like building/publishing packages are generally applicable rather than company-specific.

1. Kir Chou @ PyCon TW 2020
1
Time travel:
Let’s learn from the history of Python packaging!

2. Kir Chou
• PyCon TW Speaker 2017, 2018, 2019
• PyCon JP Speaker 2019, 2020
• SDE @ Amazon Search
2
@k1rch0u
kir.chou
note35

3. Agenda
• Motivation
• History of Python software distribution
• Learn from history
• Challenge before community’s python packaging solution
• Challenge after community’s python packaging solution
• Takeaway
• Appendix
3

4. Motivation
4

5. 5
One day, our builder Tool team announced…
Let’s migrate to idiomatic tools!

6. 6
Great! But why?

7. 7
@di_codes

8. History of Python software distribution
8

9. 9
1991
Python 1.0

10. 10
How to give/take the work to/from others?

11. How to give/take the work to/from others?
11
Give Take

12. How to give/take the work to/from others?
12
Give Take

13. 13

14. RFC822
14
1991 20052003
Python 1.0 Python 2.0
2000
PEP241 PEP345
PEP314
disutils (egg)
setuptools
2006
easy_install
PEP405
tox
virtualenv
20072008 2011
PyPI

15. 15
Metadata: PEP241 (2001), PEP314 (2003), PEP345 (2005)
Build tool: distutils, setuptools

16. 16
setuptools: setup.py
Metadata in EGG
xxx.egg-info/PKG-INFO

17. 17
PyPI (Cheese Shop): PEP301 (2002/Nov)
Package Installer: easy_install

18. 18
Isolated Environments (Build / Test / Release):
tox, virtualenv (2007), venv@PEP405 (2011)

19. 19

20. 20
1991 2011
Python 1.0
2008
Python 3.0
RFC822 PEP566
PEP459
PEP426
PEP427,440PEP376
20052003
Python 2.0
2000
PEP241 PEP345
PEP314
disutils (egg)
setuptools
2006
wheel
easy_install
bento/twine
PyPI
pip
PEP405
tox
virtualenv
2007
PEP394 PEP582
poetry
pipenv
2012 2017
PyPI
warehouse
PEP262

21. 21
Deferred Metadata 2.0: PEP426 (2012), PEP459 (2013)
Metadata 2.1: PEP566 (2017/Dec)

22. 22
PEP566: pyproject.toml

23. 23
Version & Dependency:
PEP386 (2009/Superseded), pip-tools/pip-sync (2012)
PEP440 (2013), PEP508 (2015)

24. 24
Package installer (2008): pip
Publish Tool (2013): bento / twine / flit
PyPI (2018): PyPI warehouse

25. 25
Database: PEP262 (2001/Deferred), PEP376 (2009)
Wheel: PEP425 (2012) / PEP427 (2012)

26. 26
PEP376
Database of Installed Python Distributions
xxx.dist-info/WHEEL

27. 27

28. 28
One For All: Conda / Pipenv / Poetry
PEP517 (2015), PEP518 (2016)

29. 29
Learn from history

30. 30
1. Metadata Schema Evolution

31. Evolution
31
2. Build Tools
1. Metadata Schema

32. 32
2. Build Tools
1. Metadata Schema
3. Publish Tools / Package Index
Evolution

33. 33
2. Build Tools
1. Metadata Schema
3. Publish Tools / Package Index
Evolution
4. Installer /
Deployment Tools

34. 34
2. Build Tools
1. Metadata Schema
3. Publish Tools / Package Index
5. Multiple platform
Evolution
4. Installer /
Deployment Tools

35. 35
2. Build Tools
1. Metadata Schema
3. Publish Tools / Package Index
5. Multiple platform / version
Evolution
4. Installer /
Deployment Tools

36. 36
1991
Python 1.0
wheel
pipenv
poetry
The pattern is repeated per generation
2008
Python 3.0
tox
Python 2.0
2000
disutils (egg)
setuptools
virtualenv
bento/twine
easy_install pip
PyPI
warehouse
PyPI

37. 1994 Amazon
1998 Google
37
Challenge BEFORE
community’s python packaging solution

38. 38
1991 1998
Python 1.0
Google
1994
Amazon
No Python packaging solution

39. 39
How to give/take the work to/from others?

40. Solve same problems in history
• Build tool
• build tool + dependency resolver
• Support multiple Python version
• Publish tool / Private Package Registry
• Installer / Deployment tool
• Support isolated environment
• Support multiple platforms
40

41. 41
2. Internal Build Tools
1. Internal Package Metadata
3. Internal Package Registry
4. Internal Deployment Tools

42. 42
3p packages are all handled by internal tools

43. 43

44. 44
Company
Community
X growth rate
X
Community solutions grow faster now!
~2012

45. Packaging problem is mostly general!
• Build System
• What is a Python package?
• How does build tool resolve Python’s dependencies?
• How to build a Python package?
• Publish System / Private Package Registry
• How to publish your Python package?
• How to download other Python’s dependencies?
• Deployment System
• How to isolate your Python environment?
• Which platform does the Python package support?
45

46. Shift to community solution

47. 47
Challenge AFTER
community’s python packaging solution

48. 1. Ecosystem
48
Community’s solution Company’s solution
Community’s solution Company’s solution
one way
two way

49. 49

50. 50
Cloud Services
Business
Cloud Service
Provider
Developer
Community
Sponsorship
Pay $
Invent New solution
Adopt new community solution
One for all, all for one

51. 51
2. More standards ∝ More tools

52. 52

53. 53

54. 54
Join PyCon! Join the community!

55. 55
Review the past to understand the present
Takeaway
History &
Past Challenges
Current
Challenges
Community

56. Thanks!
56

57. Appendix
57

58. History
• Dustin Ingram @ SciPy 2018
Inside the Cheeseshop: How Python Packaging Works
• History of packaging written by Martijn Faassen
• Kenneth Reitz @ PyCon 2018
Pipenv: The Future of Python Dependency Management
• Clinton Roy @ Kiwi PyCon X (2019)
The Packaging Lifecycle with Poetry
• History of wheel written by Daniel Holth
58

59. Packaging
• Dave Forgac @ PyOhio 2015
Python Packaging from Init to Deploy
• Elana Hashman @ PyCon 2019
The Black Magic of Python Wheel
• Official Document: Packaging binary extensions (2013)
59

60. Runtime (Virtual) Environment
• Carl Meyer @ PyCon 2011
Reverse-engineering Ian Bicking's brain: inside pip and
virtualenv
• Bernat Gabor @ EuroPython 2019
Status quo of virtual environments
60

61. 61
Why did 3 big cloud platforms provide
Package Registry service
JUST recently?
2019 May 2020 June 2020 July