Download as PDF, PPTX
Uploaded byKir Chou
Time travel: Let’s learn from the history of Python packaging!
This document provides a history of Python packaging and discusses lessons learned. It covers: 1. The evolution of Python packaging standards and tools from 1991 to present, including changes to metadata schemas, build tools, package publishing/indexing, installers, and support for multiple platforms/versions. 2. Challenges that existed before and after the community developed packaging solutions, such as companies initially developing their own internal tools and later adopting community standards. 3. How the community's open solutions grew faster than any individual company could achieve, and how problems like building/publishing packages are generally applicable rather than company-specific.
More Related Content
Similar to Time travel: Let’s learn from the history of Python packaging!
Time travel: Let’s learn from the history of Python packaging!
- 1. Kir Chou @PyCon TW 2020 1 Time travel: Let’s learn from the history of Python packaging!
- 2. Kir Chou • PyConTW Speaker 2017, 2018, 2019 • PyCon JP Speaker 2019, 2020 • SDE @ Amazon Search 2 @k1rch0u kir.chou note35
- 3. Agenda • Motivation • Historyof Python software distribution • Learn from history • Challenge before community’s python packaging solution • Challenge after community’s python packaging solution • Takeaway • Appendix 3
- 4.
- 5. 5 One day, ourbuilder Tool team announced… Let’s migrate to idiomatic tools!
- 6.
- 7.
- 8. History of Pythonsoftware distribution 8
- 9.
- 10. 10 How to give/takethe work to/from others?
- 11. How to give/takethe work to/from others? 11 Give Take
- 12. How to give/takethe work to/from others? 12 Give Take
- 13.
- 14. RFC822 14 1991 20052003 Python 1.0Python 2.0 2000 PEP241 PEP345 PEP314 disutils (egg) setuptools 2006 easy_install PEP405 tox virtualenv 20072008 2011 PyPI
- 15. 15 Metadata: PEP241 (2001),PEP314 (2003), PEP345 (2005) Build tool: distutils, setuptools
- 16. 16 setuptools: setup.py Metadata inEGG xxx.egg-info/PKG-INFO
- 17. 17 PyPI (Cheese Shop):PEP301 (2002/Nov) Package Installer: easy_install
- 18. 18 Isolated Environments (Build/ Test / Release): tox, virtualenv (2007), venv@PEP405 (2011)
- 19.
- 20. 20 1991 2011 Python 1.0 2008 Python3.0 RFC822 PEP566 PEP459 PEP426 PEP427,440PEP376 20052003 Python 2.0 2000 PEP241 PEP345 PEP314 disutils (egg) setuptools 2006 wheel easy_install bento/twine PyPI pip PEP405 tox virtualenv 2007 PEP394 PEP582 poetry pipenv 2012 2017 PyPI warehouse PEP262
- 21. 21 Deferred Metadata 2.0:PEP426 (2012), PEP459 (2013) Metadata 2.1: PEP566 (2017/Dec)
- 22.
- 23. 23 Version & Dependency: PEP386(2009/Superseded), pip-tools/pip-sync (2012) PEP440 (2013), PEP508 (2015)
- 24. 24 Package installer (2008):pip Publish Tool (2013): bento / twine / flit PyPI (2018): PyPI warehouse
- 25. 25 Database: PEP262 (2001/Deferred),PEP376 (2009) Wheel: PEP425 (2012) / PEP427 (2012)
- 26. 26 PEP376 Database of InstalledPython Distributions xxx.dist-info/WHEEL
- 27.
- 28. 28 One For All:Conda / Pipenv / Poetry PEP517 (2015), PEP518 (2016)
- 29.
- 30. 30 1. Metadata SchemaEvolution
- 31. Evolution 31 2. Build Tools 1.Metadata Schema
- 32. 32 2. Build Tools 1.Metadata Schema 3. Publish Tools / Package Index Evolution
- 33. 33 2. Build Tools 1.Metadata Schema 3. Publish Tools / Package Index Evolution 4. Installer / Deployment Tools
- 34. 34 2. Build Tools 1.Metadata Schema 3. Publish Tools / Package Index 5. Multiple platform Evolution 4. Installer / Deployment Tools
- 35. 35 2. Build Tools 1.Metadata Schema 3. Publish Tools / Package Index 5. Multiple platform / version Evolution 4. Installer / Deployment Tools
- 36. 36 1991 Python 1.0 wheel pipenv poetry The patternis repeated per generation 2008 Python 3.0 tox Python 2.0 2000 disutils (egg) setuptools virtualenv bento/twine easy_install pip PyPI warehouse PyPI
- 37. 1994 Amazon 1998 Google 37 ChallengeBEFORE community’s python packaging solution
- 38. 38 1991 1998 Python 1.0 Google 1994 Amazon NoPython packaging solution
- 39. 39 How to give/takethe work to/from others?
- 40. Solve same problemsin history • Build tool • build tool + dependency resolver • Support multiple Python version • Publish tool / Private Package Registry • Installer / Deployment tool • Support isolated environment • Support multiple platforms 40
- 41. 41 2. Internal BuildTools 1. Internal Package Metadata 3. Internal Package Registry 4. Internal Deployment Tools
- 42. 42 3p packages areall handled by internal tools
- 43.
- 44. 44 Company Community X growth rate X Communitysolutions grow faster now! ~2012
- 45. Packaging problem ismostly general! • Build System • What is a Python package? • How does build tool resolve Python’s dependencies? • How to build a Python package? • Publish System / Private Package Registry • How to publish your Python package? • How to download other Python’s dependencies? • Deployment System • How to isolate your Python environment? • Which platform does the Python package support? 45
- 46. Shift to communitysolution
- 47. 47 Challenge AFTER community’s pythonpackaging solution
- 48. 1. Ecosystem 48 Community’s solutionCompany’s solution Community’s solution Company’s solution one way two way
- 49.
- 50. 50 Cloud Services Business Cloud Service Provider Developer Community Sponsorship Pay$ Invent New solution Adopt new community solution One for all, all for one
- 51. 51 2. More standards∝ More tools
- 52.
- 53.
- 54. 54 Join PyCon! Jointhe community!
- 55. 55 Review the pastto understand the present Takeaway History & Past Challenges Current Challenges Community
- 56.
- 57.
- 58. History • Dustin Ingram@ SciPy 2018 Inside the Cheeseshop: How Python Packaging Works • History of packaging written by Martijn Faassen • Kenneth Reitz @ PyCon 2018 Pipenv: The Future of Python Dependency Management • Clinton Roy @ Kiwi PyCon X (2019) The Packaging Lifecycle with Poetry • History of wheel written by Daniel Holth 58
- 59. Packaging • Dave Forgac@ PyOhio 2015 Python Packaging from Init to Deploy • Elana Hashman @ PyCon 2019 The Black Magic of Python Wheel • Official Document: Packaging binary extensions (2013) 59
- 60. Runtime (Virtual) Environment •Carl Meyer @ PyCon 2011 Reverse-engineering Ian Bicking's brain: inside pip and virtualenv • Bernat Gabor @ EuroPython 2019 Status quo of virtual environments 60
- 61. 61 Why did 3big cloud platforms provide Package Registry service JUST recently? 2019 May 2020 June 2020 July