SlideShare a Scribd company logo
1 of 12
Download to read offline
Making from-source builds straight
from PyPI reliable at long last
Ralf Gommers
Qshare, 28 November 2023
Teaser - building top 150 PyPI packages
in clean containers - no GCC, make, or python-dev
Prototype of a “mini build farm”:
113 pure Python packages (trivial to build)
37 packages with native code:
See https://github.com/rgommers/external-deps-build for more results
Current state of affairs
When a package has any non-PyPI dependencies, things
tend to break down quickly
Critical info on what is needed for a successful build of a
package are (at best) hidden in its docs and CI scripts
Builds tend to fail halfway through with cryptic errors
Current state of affairs
How do we get out of this mess?
1. Allow package authors to express external
dependencies in metadata
2. Allow build front ends to check for and (optionally)
install external dependencies
3. Enable build systems to check for build
dependencies in more reliable ways
4. Enable build systems to build their own deps if not
already installed
🖺 PEP 725
🖺 PEP xxx
🛠pyproject-external
🛠 pkg-config & co
🛠 Meson subprojects
PEP 725 - external metadata in pyproject.toml
Example 1: Pillow
pyproject.toml now
what is added after PEP 725
PEP 725 - external metadata in pyproject.toml
Example 2: SciPy
pyproject.toml now what is added after PEP 725
PEP xxx - name mapping mechanism
Example, flow of actions for:
Build frontend: Pip
Build backend: meson-python
System package manager: APT
Better dependency detection
How we did this in the distutils/setuptools age:
try_compile(code_snippet,
library_dirs=lib_dirs,
include_dirs=incdirs)
Problems with that approach: just blindly checks for a library and headers,
results in pass/fail only:
- No info about detected version
- Need to try a set of hardcoded paths, which may pick up completely
wrong versions
- We have to guess what compile flags to use as well
Better dependency detection
How this should work instead:
1. Each dependency which provides library/headers should install
metadata files and/or a config script about:
a. what it provides (name, version)
b. how to use it (compile and link flags)
2. The build system queries this metadata with pkg-config, CMake, or its
own built-in detection code.
a. The try_compile approach can still be used, as a last resort.
Main challenge: actually getting all this metadata installed
Fallback builds & making distros happy
● Detect dependencies only via reliable metadata querying,
● If not found, build it from source
○ This ensures we can enforce version constraints, and use the same
compilers and build flags as for our own packages
○ It also allows us to unvendor easily - this makes distro packagers
quite happy (and is good for maintenance)
SciPy example (HiGHS):
Demo: py-show & pkg manager interop
A prototype name mapping tool with a CLI plus a
“mini build farm”, in <500 LoC

More Related Content

Similar to Reliable from-source builds (Qshare 28 Nov 2023).pdf

Conda: A Cross-Platform Package Manager for Any Binary Distribution (SciPy 2014)
Conda: A Cross-Platform Package Manager for Any Binary Distribution (SciPy 2014)Conda: A Cross-Platform Package Manager for Any Binary Distribution (SciPy 2014)
Conda: A Cross-Platform Package Manager for Any Binary Distribution (SciPy 2014)
Aaron Meurer
 

Similar to Reliable from-source builds (Qshare 28 Nov 2023).pdf (20)

Apache Bigtop and ARM64 / AArch64 - Empowering Big Data Everywhere
Apache Bigtop and ARM64 / AArch64 - Empowering Big Data EverywhereApache Bigtop and ARM64 / AArch64 - Empowering Big Data Everywhere
Apache Bigtop and ARM64 / AArch64 - Empowering Big Data Everywhere
 
Supply Chainsaw
Supply ChainsawSupply Chainsaw
Supply Chainsaw
 
Containerize your Blackbox tests
Containerize your Blackbox testsContainerize your Blackbox tests
Containerize your Blackbox tests
 
Lean Drupal Repositories with Composer and Drush
Lean Drupal Repositories with Composer and DrushLean Drupal Repositories with Composer and Drush
Lean Drupal Repositories with Composer and Drush
 
So. many. vulnerabilities. Why are containers such a mess and what to do abou...
So. many. vulnerabilities. Why are containers such a mess and what to do abou...So. many. vulnerabilities. Why are containers such a mess and what to do abou...
So. many. vulnerabilities. Why are containers such a mess and what to do abou...
 
ICONUK 2015 - Gradle Up!
ICONUK 2015 - Gradle Up!ICONUK 2015 - Gradle Up!
ICONUK 2015 - Gradle Up!
 
Puppet Camp LA 2015: Package Managers and Puppet (Beginner)
Puppet Camp LA 2015: Package Managers and Puppet (Beginner)Puppet Camp LA 2015: Package Managers and Puppet (Beginner)
Puppet Camp LA 2015: Package Managers and Puppet (Beginner)
 
Puppet Camp LA 2/19/2015
Puppet Camp LA 2/19/2015Puppet Camp LA 2/19/2015
Puppet Camp LA 2/19/2015
 
Bringing-it-all-together-overview-of-rpm-packaging-in-fedora
Bringing-it-all-together-overview-of-rpm-packaging-in-fedoraBringing-it-all-together-overview-of-rpm-packaging-in-fedora
Bringing-it-all-together-overview-of-rpm-packaging-in-fedora
 
TDD for joomla extensions
TDD for joomla extensionsTDD for joomla extensions
TDD for joomla extensions
 
LIGGGHTS installation-guide
LIGGGHTS installation-guideLIGGGHTS installation-guide
LIGGGHTS installation-guide
 
Managing Perl Installations: A SysAdmin's View
Managing Perl Installations: A SysAdmin's ViewManaging Perl Installations: A SysAdmin's View
Managing Perl Installations: A SysAdmin's View
 
PURL and vers: The Mostly Universal Package URL and Version Ranges Identifier...
PURL and vers: The Mostly Universal Package URL and Version Ranges Identifier...PURL and vers: The Mostly Universal Package URL and Version Ranges Identifier...
PURL and vers: The Mostly Universal Package URL and Version Ranges Identifier...
 
Conda: A Cross-Platform Package Manager for Any Binary Distribution (SciPy 2014)
Conda: A Cross-Platform Package Manager for Any Binary Distribution (SciPy 2014)Conda: A Cross-Platform Package Manager for Any Binary Distribution (SciPy 2014)
Conda: A Cross-Platform Package Manager for Any Binary Distribution (SciPy 2014)
 
Effizientere WordPress-Plugin-Entwicklung mit Softwaretests
Effizientere WordPress-Plugin-Entwicklung mit SoftwaretestsEffizientere WordPress-Plugin-Entwicklung mit Softwaretests
Effizientere WordPress-Plugin-Entwicklung mit Softwaretests
 
Deploying software at Scale
Deploying software at ScaleDeploying software at Scale
Deploying software at Scale
 
Packaging perl (LPW2010)
Packaging perl (LPW2010)Packaging perl (LPW2010)
Packaging perl (LPW2010)
 
OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...
OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...
OSDC 2016 - Continous Integration in Data Centers - Further 3 Years later by ...
 
CMake - Introduction and best practices
CMake - Introduction and best practicesCMake - Introduction and best practices
CMake - Introduction and best practices
 
Ansible Workshop for Pythonistas
Ansible Workshop for PythonistasAnsible Workshop for Pythonistas
Ansible Workshop for Pythonistas
 

More from Ralf Gommers

More from Ralf Gommers (13)

Parallelism in a NumPy-based program
Parallelism in a NumPy-based programParallelism in a NumPy-based program
Parallelism in a NumPy-based program
 
The road ahead for scientific computing with Python
The road ahead for scientific computing with PythonThe road ahead for scientific computing with Python
The road ahead for scientific computing with Python
 
Python array API standardization - current state and benefits
Python array API standardization - current state and benefitsPython array API standardization - current state and benefits
Python array API standardization - current state and benefits
 
Building SciPy kernels with Pythran
Building SciPy kernels with PythranBuilding SciPy kernels with Pythran
Building SciPy kernels with Pythran
 
Standardizing on a single N-dimensional array API for Python
Standardizing on a single N-dimensional array API for PythonStandardizing on a single N-dimensional array API for Python
Standardizing on a single N-dimensional array API for Python
 
Strengthening NumPy's foundations - growing beyond code
Strengthening NumPy's foundations - growing beyond codeStrengthening NumPy's foundations - growing beyond code
Strengthening NumPy's foundations - growing beyond code
 
PyData NYC whatsnew NumPy-SciPy 2019
PyData NYC whatsnew NumPy-SciPy 2019PyData NYC whatsnew NumPy-SciPy 2019
PyData NYC whatsnew NumPy-SciPy 2019
 
Inside NumPy: preparing for the next decade
Inside NumPy: preparing for the next decadeInside NumPy: preparing for the next decade
Inside NumPy: preparing for the next decade
 
The evolution of array computing in Python
The evolution of array computing in PythonThe evolution of array computing in Python
The evolution of array computing in Python
 
__array_function__ conceptual design & related concepts
__array_function__ conceptual design & related concepts__array_function__ conceptual design & related concepts
__array_function__ conceptual design & related concepts
 
NumPy Roadmap presentation at NumFOCUS Forum
NumPy Roadmap presentation at NumFOCUS ForumNumPy Roadmap presentation at NumFOCUS Forum
NumPy Roadmap presentation at NumFOCUS Forum
 
NumFOCUS_Summit2018_Roadmaps_session
NumFOCUS_Summit2018_Roadmaps_sessionNumFOCUS_Summit2018_Roadmaps_session
NumFOCUS_Summit2018_Roadmaps_session
 
SciPy 1.0 and Beyond - a Story of Community and Code
SciPy 1.0 and Beyond - a Story of Community and CodeSciPy 1.0 and Beyond - a Story of Community and Code
SciPy 1.0 and Beyond - a Story of Community and Code
 

Recently uploaded

Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
 
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
drm1699
 
Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
 

Recently uploaded (20)

Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
Workshop -  Architecting Innovative Graph Applications- GraphSummit MilanWorkshop -  Architecting Innovative Graph Applications- GraphSummit Milan
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
 
Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
Abortion Clinic in Midrand [(+27832195400*)]🏥Safe Abortion Pills In Midrand |...
 
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfAzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
 
Artyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptxArtyushina_Guest lecture_YorkU CS May 2024.pptx
Artyushina_Guest lecture_YorkU CS May 2024.pptx
 
GraphSummit Milan - Neo4j: The Art of the Possible with Graph
GraphSummit Milan - Neo4j: The Art of the Possible with GraphGraphSummit Milan - Neo4j: The Art of the Possible with Graph
GraphSummit Milan - Neo4j: The Art of the Possible with Graph
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
 
Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
 
Lessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdfLessons Learned from Building a Serverless Notifications System.pdf
Lessons Learned from Building a Serverless Notifications System.pdf
 
Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Rustenburg [(+27832195400*)] 🏥 Women's Abortion Clinic i...
 
Transformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with LinksTransformer Neural Network Use Cases with Links
Transformer Neural Network Use Cases with Links
 
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
Abortion Pill Prices Aliwal North ](+27832195400*)[ 🏥 Women's Abortion Clinic...
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
 
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
Wired_2.0_CREATE YOUR ULTIMATE LEARNING ENVIRONMENT_JCON_16052024
 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the Situation
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Concepts
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeCon
 

Reliable from-source builds (Qshare 28 Nov 2023).pdf

  • 1. Making from-source builds straight from PyPI reliable at long last Ralf Gommers Qshare, 28 November 2023
  • 2. Teaser - building top 150 PyPI packages in clean containers - no GCC, make, or python-dev Prototype of a “mini build farm”: 113 pure Python packages (trivial to build) 37 packages with native code: See https://github.com/rgommers/external-deps-build for more results
  • 3. Current state of affairs When a package has any non-PyPI dependencies, things tend to break down quickly Critical info on what is needed for a successful build of a package are (at best) hidden in its docs and CI scripts Builds tend to fail halfway through with cryptic errors
  • 4. Current state of affairs
  • 5. How do we get out of this mess? 1. Allow package authors to express external dependencies in metadata 2. Allow build front ends to check for and (optionally) install external dependencies 3. Enable build systems to check for build dependencies in more reliable ways 4. Enable build systems to build their own deps if not already installed 🖺 PEP 725 🖺 PEP xxx 🛠pyproject-external 🛠 pkg-config & co 🛠 Meson subprojects
  • 6. PEP 725 - external metadata in pyproject.toml Example 1: Pillow pyproject.toml now what is added after PEP 725
  • 7. PEP 725 - external metadata in pyproject.toml Example 2: SciPy pyproject.toml now what is added after PEP 725
  • 8. PEP xxx - name mapping mechanism Example, flow of actions for: Build frontend: Pip Build backend: meson-python System package manager: APT
  • 9. Better dependency detection How we did this in the distutils/setuptools age: try_compile(code_snippet, library_dirs=lib_dirs, include_dirs=incdirs) Problems with that approach: just blindly checks for a library and headers, results in pass/fail only: - No info about detected version - Need to try a set of hardcoded paths, which may pick up completely wrong versions - We have to guess what compile flags to use as well
  • 10. Better dependency detection How this should work instead: 1. Each dependency which provides library/headers should install metadata files and/or a config script about: a. what it provides (name, version) b. how to use it (compile and link flags) 2. The build system queries this metadata with pkg-config, CMake, or its own built-in detection code. a. The try_compile approach can still be used, as a last resort. Main challenge: actually getting all this metadata installed
  • 11. Fallback builds & making distros happy ● Detect dependencies only via reliable metadata querying, ● If not found, build it from source ○ This ensures we can enforce version constraints, and use the same compilers and build flags as for our own packages ○ It also allows us to unvendor easily - this makes distro packagers quite happy (and is good for maintenance) SciPy example (HiGHS):
  • 12. Demo: py-show & pkg manager interop A prototype name mapping tool with a CLI plus a “mini build farm”, in <500 LoC