Modern binary build systems have made shipping binary packages for Python much easier than ever before. This talk discusses three of the most popular build systems for Python packages using the new standards developed for packaging.
SciPy22 - Building binary extensions with pybind11, scikit build, and cibuild...Henry Schreiner
Building binary extensions is easier than ever thanks to several key libraries. Pybind11 provides a natural C++ language for extensions without requiring pre-processing or special dependencies. Scikit-build ties the premier C++ build system, CMake, into the Python extension build process. And cibuildwheel makes it easy to build highly compatible wheels for over 80 different platforms using CI or on your local machine. We will look at advancements to all three libraries over the last year, as well as future plans.
Scikit-HEP has grown rapidly over the last few years, not just to serve the needs of the High Energy Physics (HEP) community, but in many ways, the Python ecosystem at large. AwkwardArray, boost-histogram/hist, and iMinuit are examples of libraries that are used beyond the original HEP focus. In this talk, we will look key packages in the ecosystem, at how the collection of 30+ packages was developed and maintained, and the software ecosystem contributions made to packages like cibuildwheel, pybind11, nox, scikit-build, build, and pipx that support this effort, and the Scikit-HEP developer pages, and initial WebAssembly support.
Euro python2011 High Performance PythonIan Ozsvald
I ran this as a 4 hour tutorial at EuroPython 2011 to teach High Performance Python coding.
Techniques covered include bottleneck analysis by profiling, bytecode analysis, converting to C using Cython and ShedSkin, use of the numerical numpy library and numexpr, multi-core and multi-machine parallelisation and using CUDA GPUs.
Write-up with 49 page PDF report: http://ianozsvald.com/2011/06/29/high-performance-python-tutorial-v0-1-from-my-4-hour-tutorial-at-europython-2011/
Talk at PyCon2022 over building binary packages for Python. Covers an overview and an in-depth look into pybind11 for binding, scikit-build for creating the build, and build & cibuildwheel for making the binaries that can be distributed on PyPI.
SciPy22 - Building binary extensions with pybind11, scikit build, and cibuild...Henry Schreiner
Building binary extensions is easier than ever thanks to several key libraries. Pybind11 provides a natural C++ language for extensions without requiring pre-processing or special dependencies. Scikit-build ties the premier C++ build system, CMake, into the Python extension build process. And cibuildwheel makes it easy to build highly compatible wheels for over 80 different platforms using CI or on your local machine. We will look at advancements to all three libraries over the last year, as well as future plans.
Scikit-HEP has grown rapidly over the last few years, not just to serve the needs of the High Energy Physics (HEP) community, but in many ways, the Python ecosystem at large. AwkwardArray, boost-histogram/hist, and iMinuit are examples of libraries that are used beyond the original HEP focus. In this talk, we will look key packages in the ecosystem, at how the collection of 30+ packages was developed and maintained, and the software ecosystem contributions made to packages like cibuildwheel, pybind11, nox, scikit-build, build, and pipx that support this effort, and the Scikit-HEP developer pages, and initial WebAssembly support.
Euro python2011 High Performance PythonIan Ozsvald
I ran this as a 4 hour tutorial at EuroPython 2011 to teach High Performance Python coding.
Techniques covered include bottleneck analysis by profiling, bytecode analysis, converting to C using Cython and ShedSkin, use of the numerical numpy library and numexpr, multi-core and multi-machine parallelisation and using CUDA GPUs.
Write-up with 49 page PDF report: http://ianozsvald.com/2011/06/29/high-performance-python-tutorial-v0-1-from-my-4-hour-tutorial-at-europython-2011/
Talk at PyCon2022 over building binary packages for Python. Covers an overview and an in-depth look into pybind11 for binding, scikit-build for creating the build, and build & cibuildwheel for making the binaries that can be distributed on PyPI.
Digital RSE: automated code quality checks - RSE group meetingHenry Schreiner
Given at a local RSE group meeting. Covers code quality practices, focusing on Python but over multiple languages, with useful tools highlighted throughout.
Slides for the Cluj.py meetup where we explored the inner workings of CPython, the reference implementation of Python. Includes examples of writing a C extension to Python, and introduces Cython - ultimately the sanest way of writing C extensions.
Also check out the code samples on GitHub: https://github.com/trustyou/meetups/tree/master/python-c
Keynote talk at PyCon Estonia 2019 where I discuss how to extend CPython and how that has led to a robust ecosystem around Python. I then discuss the need to define and build a Python extension language I later propose as EPython on OpenTeams: https://openteams.com/initiatives/2
If you're tired of dealing with load balancing, routing and cloud monitoring and would like to focus on just lambda function, list comprehension or class-based view, then this is the talk for you!
Heroku is the Platform as a Service (PaaS) platform that allows developers to create, run and manage applications entirely in the cloud. To be provided as a service not only the hardware, but also the platform that abstracts the hardware itself and allows you to take advantage of features that allow you to achieve automatic balancing, deployment management and more.
The advantage for the user is to focus exclusively on the development of the application without getting lost in the analysis of problems related to the environment in which it must be distributed, thus obtaining the necessary scalability and reliability.
Python for Delphi (P4D) is a set of free components that wrap up the Python DLL into Delphi and Lazarus (FPC). They let you easily execute Python scripts, create new Python modules and new Python types. You can create Python extensions as DLLs and much more like scripting. P4D provides different levels of functionality: Low-level access to the python API High-level bi-directional interaction with Python Access to Python objects using Delphi custom variants (VarPyth.pas).
In this slides, the author demonstrates many software development practices in Python. Including: runtime environment setup, source code management, version control, unit test, coding convention, code duplication, documentation and automation.
[GSoC 2017] gopy: Updating gopy to support Python3 and PyPyDong-hee Na
gopy is an excellent tool which generates (and compiles) a CPython extension module from a go package. And I hope more developers could make full use of gopy to migrate their go code into python code. To make gopy more advanced, It is necessary to provide APIs for various Python compiler versions, such as CPython 2/3 and PyPy. This can be improved with CFFI or ctypes. Moreover, many go’s implementations/features are not yet implemented in gopy. So we need to implement implementations such as slices, interfaces, and maps in the go.
My goal is to update gopy by using CFFI to support Python3 and PyPy and write detailed documents
Digital RSE: automated code quality checks - RSE group meetingHenry Schreiner
Given at a local RSE group meeting. Covers code quality practices, focusing on Python but over multiple languages, with useful tools highlighted throughout.
Slides for the Cluj.py meetup where we explored the inner workings of CPython, the reference implementation of Python. Includes examples of writing a C extension to Python, and introduces Cython - ultimately the sanest way of writing C extensions.
Also check out the code samples on GitHub: https://github.com/trustyou/meetups/tree/master/python-c
Keynote talk at PyCon Estonia 2019 where I discuss how to extend CPython and how that has led to a robust ecosystem around Python. I then discuss the need to define and build a Python extension language I later propose as EPython on OpenTeams: https://openteams.com/initiatives/2
If you're tired of dealing with load balancing, routing and cloud monitoring and would like to focus on just lambda function, list comprehension or class-based view, then this is the talk for you!
Heroku is the Platform as a Service (PaaS) platform that allows developers to create, run and manage applications entirely in the cloud. To be provided as a service not only the hardware, but also the platform that abstracts the hardware itself and allows you to take advantage of features that allow you to achieve automatic balancing, deployment management and more.
The advantage for the user is to focus exclusively on the development of the application without getting lost in the analysis of problems related to the environment in which it must be distributed, thus obtaining the necessary scalability and reliability.
Python for Delphi (P4D) is a set of free components that wrap up the Python DLL into Delphi and Lazarus (FPC). They let you easily execute Python scripts, create new Python modules and new Python types. You can create Python extensions as DLLs and much more like scripting. P4D provides different levels of functionality: Low-level access to the python API High-level bi-directional interaction with Python Access to Python objects using Delphi custom variants (VarPyth.pas).
In this slides, the author demonstrates many software development practices in Python. Including: runtime environment setup, source code management, version control, unit test, coding convention, code duplication, documentation and automation.
[GSoC 2017] gopy: Updating gopy to support Python3 and PyPyDong-hee Na
gopy is an excellent tool which generates (and compiles) a CPython extension module from a go package. And I hope more developers could make full use of gopy to migrate their go code into python code. To make gopy more advanced, It is necessary to provide APIs for various Python compiler versions, such as CPython 2/3 and PyPy. This can be improved with CFFI or ctypes. Moreover, many go’s implementations/features are not yet implemented in gopy. So we need to implement implementations such as slices, interfaces, and maps in the go.
My goal is to update gopy by using CFFI to support Python3 and PyPy and write detailed documents
Similar to Modern binary build systems - PyCon 2024 (20)
PyCon 2022 -Scikit-HEP Developer Pages: Guidelines for modern packagingHenry Schreiner
This was a PyCon 2022 lightning talk over the Scikit-HEP developer pages. It highlights best practices and guides shown there, and the quick package creation cookiecutter. And finally it demos the Pyodide WebAssembly app embedded into the Scikit-HEP developer pages!
Atelier - Innover avec l’IA Générative et les graphes de connaissancesNeo4j
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement.
Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Crescat
Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry.
Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events.
With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use.
Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements.
If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io
Artificia Intellicence and XPath Extension FunctionsOctavian Nadolu
The purpose of this presentation is to provide an overview of how you can use AI from XSLT, XQuery, Schematron, or XML Refactoring operations, the potential benefits of using AI, and some of the challenges we face.
OpenMetadata Community Meeting - 5th June 2024OpenMetadata
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
E-commerce Application Development Company.pdfHornet Dynamics
Your business can reach new heights with our assistance as we design solutions that are specifically appropriate for your goals and vision. Our eCommerce application solutions can digitally coordinate all retail operations processes to meet the demands of the marketplace while maintaining business continuity.
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Software Engineering, Software Consulting, Tech Lead, Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Transaction, Spring MVC, OpenShift Cloud Platform, Kafka, REST, SOAP, LLD & HLD.
Launch Your Streaming Platforms in MinutesRoshan Dwivedi
The claim of launching a streaming platform in minutes might be a bit of an exaggeration, but there are services that can significantly streamline the process. Here's a breakdown:
Pros of Speedy Streaming Platform Launch Services:
No coding required: These services often use drag-and-drop interfaces or pre-built templates, eliminating the need for programming knowledge.
Faster setup: Compared to building from scratch, these platforms can get you up and running much quicker.
All-in-one solutions: Many services offer features like content management systems (CMS), video players, and monetization tools, reducing the need for multiple integrations.
Things to Consider:
Limited customization: These platforms may offer less flexibility in design and functionality compared to custom-built solutions.
Scalability: As your audience grows, you might need to upgrade to a more robust platform or encounter limitations with the "quick launch" option.
Features: Carefully evaluate which features are included and if they meet your specific needs (e.g., live streaming, subscription options).
Examples of Services for Launching Streaming Platforms:
Muvi [muvi com]
Uscreen [usencreen tv]
Alternatives to Consider:
Existing Streaming platforms: Platforms like YouTube or Twitch might be suitable for basic streaming needs, though monetization options might be limited.
Custom Development: While more time-consuming, custom development offers the most control and flexibility for your platform.
Overall, launching a streaming platform in minutes might not be entirely realistic, but these services can significantly speed up the process compared to building from scratch. Carefully consider your needs and budget when choosing the best option for you.
GraphSummit Paris - The art of the possible with Graph TechnologyNeo4j
Sudhir Hasbe, Chief Product Officer, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
openEuler Case Study - The Journey to Supply Chain Security
Modern binary build systems - PyCon 2024
1. Henry Schreiner • May 17, 2024 • PyCon US
Modern binary build systems
2. About me
@henryiii
2
Bec
a
me involved in p
a
ck
a
ging vi
a
Scikit-HEP
W
a
nted to ship bin
a
ry extensions
(boost-histogr
a
m,
a
wkw
a
rd, iminuit)
Joined forces with cibuildwheel
Currently m
a
int
a
in 34+ p
a
ck
a
ges
21 in top 8,000 on PyPI
pybind11 (python_example, cmake_example, scikit_build_example) •
cibuildwheel • build • pipx • pyproject-metadata •
scikit-build (core, cmake, ninja, moderncmakedomain, example-projects) •
nox • validate-pyproject(-schema-store) • pytest GHA annotate-failures •
Plumbum •
fl
ake8-errmsg • check-sdist •
boost-histogram • Hist • UHI • Vector • GooFit • Particle • DecayLanguage •
uproot-browser • Conda-Forge ROOT •
Scienti
fi
c-Python/cookie • repo-review • meson-python •
POVM • hypernewsviewer •
CLI11 • beautifulhugo • Jekyll-Indico
https://iscinumpy.dev
(Slides will be posted here l
a
ter)
3. Bene
f
its?
3
Fast
Wrap existing libraries
Cross-language
Ship existing CLI tools on PyPI
Can release GIL
Of the top 8,000 p
a
ck
a
ges on PyPI:
6,433 ship
a
single wheel
896 ship multiple wheels (prob
a
bly bin
a
ries!)
671 don't ship
a
wheel (unknown)
Over 10%!
charset-normalizer 367 M
pyyaml 279 M
numpy 246 M
cryptography 238 M
c
f
207 M
pandas 195 M
protobuf 174 M
markupsafe 151 M
wrapt 120 M
pyarrow 110 M
sqlalchemy 104 M
aiohttp 102 M
scipy 101 M
multidict 97 M
psutil 96 M
yarl 95 M
frozenlist 90 M
pillow 89 M
grpcio 88 M
greenlet 88 M
pydantic-core 83 M
4. Pure Python packages
4
Build con
fi
guration One wheel, one sdist
Binding API Build con
fi
guration Many wheels
pybind11 (C++11)
n
a
nobind (C++17)
Cython (C, C++)
SWIG (C, C++)
PyO3 (Rust)
N
a
tive (C)
cibuildwheel
m
a
turin-
a
ction (Rust)
Scikit-build-core
meson-python
m
a
turin
Binary packages
h
a
tchling
setuptools
poetry-core
build
hynek/build-
a
nd-inspect-python-p
a
ck
a
ge
f
lit-core
pdm-b
a
ckend
...
7. cibuildwheel
New features
• CPython 3.13b1 (opt-in)
• Inherit for overrides (except con
f
ig-settings for now)
• Musllinux 1.2 def
a
ult
• Support Apple Silicon runners on GHA
• Addition
a
l
f
l
a
gs to build frontend
• Better reproducible build support
• E
a
sier loc
a
l runs
7
WIP:
Fix con
f
ig-settings inherit override bug
Free-thre
a
ded support (in m
a
nylinux now!)
Investig
a
ting uv support
8. Before PEP 517 (2017)
Setuptools / distutils
8
from setuptools import Extension, setup
setup(
name = "mylib",
version = "1.0.0",
ext_modules = [
Extension(
name = "mylib.foo",
sources = ["foo.c"],
),
]
)
Simple! Unless you need…
Third p
a
rty dependencies
Compiler speci
f
ic
f
l
a
gs, like C++ version
P
a
r
a
llel (
f
ile) compiles
C
a
ching / sm
a
rt recompiles
bdist_wheel customiz
a
tions (no public API)
Fortr
a
n
Other compilers
IDE support
Tooling integr
a
tion (debuggers, etc)
Cross-compil
a
tion
Setuptools isn’t trying to
a
dd fe
a
tures to support this
So everyone h
a
s to reinvent the wheel
NumPy h
a
d 13,000 LoC for building!
(MANIFEST.in omitted)
9. Before PEP 517 (2017)
Extending setuptools / distutils
9
Ex
a
mples of setuptools-b
a
sed builders
Scikit-build (cl
a
ssic)
setuptools-rust
pybind11’s setup_helper
cython (integr
a
ted)
Distutils -> setuptools h
a
s l
a
yered complexity
h
a
rd to debug
e
a
sy to bre
a
k on upd
a
te
h
a
rd to extend
10. Modern build system
Hooks for build backends
10
Simple st
a
nd
a
rdized API
Supports inst
a
lling build requirements (dyn
a
mic
a
lly too)
B
a
ckend is responsible for b
a
sic
a
lly everything
Edit
a
ble inst
a
lls
a
dded l
a
ter
11. Build backend hooks
11
Dyn
a
mic
a
lly request
dependencies per st
a
ge
get_requires_for_build_sdist(…)
get_requires_for_build_wheel(…)
get_requires_for_build_editable(…)
build_sdist(…)
build_wheel(…)
build_editable(…)
prepare_metadata_for_build_wheel()
Given source
a
nd con
f
ig,
m
a
ke *.t
a
r.gz
f
ile
And .whl
f
ile
And “edit
a
ble” vi
a
.whl
f
ile
Option
a
l get met
a
d
a
t
a
without build
12. Rise of PEP 517 build backends
The new era of Python packaging
12
Now we h
a
ve bin
a
ry ones too!
enscons (2017)
m
a
turin (2019)
meson-python (2021)
scikit-build-core (2022)
Pure-Python b
a
ckends were
f
irst:
f
lit-core (2017*)
poetry-core (2018*)
pdm-b
a
ckend (2020*)
h
a
tchling (2022)
setuptools (2022)
Speci
a
l mention: enscons w
a
s the
f
irst PEP 517 build b
a
ckend for bin
a
ries!
(We’ll focus on the other three tod
a
y)
13. History of meson-python
13
SciPy st
a
rted the project in prep
a
r
a
tion of Python 3.12
Other m
a
jor Scienti
f
ic Python libr
a
ries joined
NumPy went from 13K to 2K LoC for building
(Not counting forking meson (100K)
a
nd vendoring it)
15. Meson-Python
Features
15
C
a
n skip PEP 621
a
nd use Meson con
f
ig for quick projects
Sever
a
l options c
a
n be speci
f
ied in pyproject.toml or vi
a
con
f
ig-settings
Developed rebuild support for edit
a
ble inst
a
lls
Ninj
a
p
a
ck
a
ge
a
dded only if missing
16. Scikit-build
A family of tools
• Scikit-build-core
• Scikit-build (cl
a
ssic)
• Ninj
a
Python Distribution (pip inst
a
ll cm
a
ke)
• CM
a
ke Python Distribution (pip inst
a
ll ninj
a
with jobserver support)
• ModernCM
a
keDom
a
in (Sphinx plugin)
• Dyn
a
mic-Met
a
d
a
t
a
(WIP project for gener
a
l met
a
d
a
t
a
plugins)
16
Scikit-build-core funded vi
a
NSF gr
a
nt OAC 2209877
17. Scikit-build-core
CMake
17
example-project
├── example.cpp
├── pyproject.toml
└── CMakeLists.txt
[build-system]
requires = ["scikit-build-core", "pybind11"]
build-backend = "scikit_build_core.build"
[project]
name = "example"
version = "0.0.1"
cmake_minimum_required(VERSION 3.15…3.29)
project(example LANGUAGES CXX)
set(PYBIND11_NEWPYTHON ON)
find_package(pybind11 CONFIG REQUIRED)
pybind11_add_module(example example.cpp)
install(TARGETS example LIBRARY DESTINATION .)
pipx run build --installer=uv
#include <pybind11/pybind11.h>
namespace py = pybind11;
float square(float x) { return x * x; }
PYBIND11_MODULE(example, m) {
m.def("square", &square);
}
18. Scikit-build-core
Features
• Autom
a
tic inclusion of cm
a
ke/ninj
a
a
s needed
• Limited API / St
a
ble ABI
a
nd pythonless t
a
gs supported vi
a
con
f
ig option
• Support for writing out to extr
a
wheel folders (scripts, he
a
ders, d
a
t
a
, met
a
d
a
t
a
)
• Dedic
a
ted entrypoints for p
a
ck
a
ges to host CM
a
ke code
• Experiment
a
l edit
a
ble mode support, with option
a
l experiment
a
l
a
uto rebuilds on import
• Free-thre
a
ded Python support
18
19. Scikit-build-core con
f
ig system
A peek into the internals
19
D
a
t
a
cl
a
ss-b
a
sed con
f
ig system
Gener
a
tes JSONSchem
a
Gener
a
tes section in README using cog
Supports pyproject.toml, con
f
ig-settings,
a
nd environment v
a
ri
a
bles
@dataclasses.dataclass
class WheelSettings:
packages: Optional[List[str]] = None
"""
A list of packages to auto-copy into the wheel. If this is not set, it will
default to the first of ``src/<package>``, ``python/<package>``, or
``<package>`` if they exist. The prefix(s) will be stripped from the
package name inside the wheel.
"""
py_api: str = ""
"""
The Python tags. The default (empty string) will use the default Python
version. You can also set this to "cp37" to enable the CPython 3.7+ Stable
ABI / Limited API (only on CPython and if the version is sufficient,
otherwise this has no effect). Or you can set it to "py3" or "py2.py3" to
ignore Python ABI compatibility. The ABI tag is inferred from this tag.
"""
expand_macos_universal_tags: bool = False
"""
Fill out extra tags that are not required. This adds "x86_64" and "arm64"
to the list of platforms when "universal2" is used, which helps older
Pip's (before 21.0.1) find the correct wheel.
"""
21. Scikit-build-core
Dynamic metadata
21
[tool.scikit-build.metadata.version]
provider = "scikit_build_core.metadata.regex"
input = "src/mypackage/__init__.py"
[[tool.scikit-build.generate]]
location = "install"
path = "mypackage/_version.py"
template = '''
version = "${version}"
'''
Join us on scikit-build/dyn
a
mic-met
a
d
a
t
a
if you
a
re
interested in gener
a
lizing for multiple b
a
ckends!
22. Scikit-build-core
Overrides
22
[[tool.scikit-build.overrides]]
if.platform-system = "darwin"
cmake.version = ">=3.18"
Designed
a
fter cibuildwheel's overrides, inspired by mypy
M
a
tch top to bottom
C
a
n use if.
a
ny
C
a
n inherit with "
a
ppend", "prepend", or "none"
T
a
kes regex, speci
f
ier set, or bool
[[tool.scikit-build.overrides]]
if.state = "editable"
build-dir = "build"
[[tool.scikit-build.overrides]]
if.any.env.SET_FOO = true
if.any.env.FOO_SET = true
inherit.cmake.define = "append"
cmake.define.FOO = "1"
24. Scikit-build-core
Some projects
24
R
a
pids.
a
i
Added to
f
ile gener
a
tor
a
nd
deployed
a
cross
a
ll projects
[tool.scikit-build]
wheel.packages = ["zmq"]
wheel.license-files = ["licenses/LICENSE*"]
cmake.version = ">=3.15"
# only build/install the pyzmq component
cmake.targets = ["pyzmq"]
install.components = ["pyzmq"]
PyZMQ
Added to
f
ile gener
a
tor
a
nd
deployed
a
cross
a
ll projects
[tool.scikit-build]
minimum-version = "0.8"
build-dir = "build/{wheel_tag}"
cmake.version = "" # We are cmake, so don't request cmake
ninja.make-fallback = false
wheel.py-api = "py3"
wheel.expand-macos-universal-tags = true
wheel.install-dir = "cmake/data"
[[tool.scikit-build.generate]]
path = "cmake/_version.py"
template = '''
version = "${version}"
'''
cm
a
ke (python distributions)
Fun met
a
problem: we
a
re cm
a
ke!
25. Bonus: Scikit-build-core plugins
Scikit-build-core is also useful for creating plugins!
25
[build-system]
requires = ["hatchling", "scikit-build-core~=0.9.0"]
build-backend = "hatchling.build"
[project]
name = "hatchling_example"
version = "0.1.0"
[tool.hatch.build.targets.wheel.hooks.scikit-build]
experimental = true
This will
a
dd
a
CM
a
ke extension to
a
h
a
tchling project!
27. PyPI Stats
Looking at every pyproject.toml
27
tool.*:
poetry: 52183
black: 24730
pytest: 18229
isort: 18111
setuptools: 15210
mypy: 13030
ruff: 12213
coverage: 10749
setuptools_scm: 7006
hatch: 6900
pylint: 4261
pyright: 3635
flit: 3039
flake8: 2216
pdm: 1995
tox: 1577
semantic_release: 1553
cibuildwheel: 1474
codespell: 1431
maturin: 1207
poetry-dynamic-versioning: 1150
C
a
n look
a
t the most
popul
a
r tool sections:
This helped bl
a
ck
check the imp
a
ct
of
a
con
f
ig bug!
tool.black.line-length contents:
120: 5755
88: 3578
79: 2876
100: 2714
80: 834
...
C
a
n check to see wh
a
t
our users
a
re using
a
nd
setting!
*:
build-system: 136738
tool: 100937
project: 51268
options: 305
tools: 279
metadata: 202
coverage: 148
mypy: 127
requires: 119
flake8: 119
virtualenvs: 90
build-backend: 81
pytest: 76
dependencies: 67
bdist_wheel: 40
build: 36
pypi: 34
build_system: 32
dev-dependencies: 30
package: 30
bumpver: 28
manageprojects: 25
too: 25
...
Or typos:
Nothing p
a
st
here is v
a
lid!
(Use v
a
lid
a
te-pyproject)!
29. Maturin
Cargo (Rust)
29
A Rust-b
a
sed build system for C
a
rgo
example-project
├── src/lib.rs
├── Cargo.toml
└── pyproject.toml
[package]
name = "package"
version = "0.1.0"
edition = "2021"
[lib]
name = "example"
crate-type = ["cdylib"]
[dependencies.pyo3]
version = "0.21.1"
features = [
"extension-module",
"abi3-py38",
"experimental-declarative-modules",
]
use pyo3::prelude::*;
#[pymodule]
mod example {
use super::*;
#[pyfunction]
fn add(x: i64, y: i64) -> i64 {
x + y
}
}
[build-system]
requires = ["maturin~=1.5"]
build-backend = "maturin"
[project]
name = "example"
dynamic = ["version"]
pipx run build —installer=uv
maturin build --release
30. Maturin
Features
30
Written in Rust
(c
a
n build
a
p
a
ck
a
ge without running Python!)
Limited API doesn't even require Python
a
t
a
ll!
C
a
rgo's build-from-source p
a
ck
a
ging is perfect
for cross-compil
a
tion
a
nd m
a
king wheels!
CLI c
a
n cre
a
te
a
new project.
Why Rust?
Rust/C
a
rgo m
a
kes using
libr
a
ries
a
s e
a
sy
a
s Python!
Cross-compiles
a
re gre
a
t
(c
a
n
a
void most m
a
nylinux issues)
Tooling is f
a
nt
a
stic
(linter, form
a
tter, IDEs, etc)
PyO3 bindings
a
re gre
a
t
Single bin
a
ry
a
ppro
a
ch
works well for wheels
(And m
a
ybe some memory s
a
fety stu
ff
)
31. Maturin-Action
GitHub Action for Building
31
Gre
a
t
a
t cross-compiling with extensive support
(f
a
st)
Builds non-Python Rust p
a
rts once for
a
ll versions
Every wheel supported except musl s390x wheels
(some pl
a
tforms like 32-bit musl c
a
n't be done with
cibuildwheel since rust doesn't build c
a
rgo for them)
...
Compiling lexical-sort v0.3.1
Compiling taplo v0.13.0
Compiling pep508_rs v0.6.0
Compiling pyproject-fmt-rust v1.1.1 (/home/runner/...
Finished release [optimized] target(s) in 13.53s
📦 Including files matching "rust-toolchain.toml"
📦 Built wheel for abi3 Python ≥ 3.8 to dist/...
⚠ Warning: PyPy does not yet support abi3 so the build...
Compiling pyo3-build-config v0.21.2
Compiling pyo3-macros-backend v0.21.2
Compiling pyo3-ffi v0.21.2
Compiling pyo3 v0.21.2
Compiling pyo3-macros v0.21.2
Compiling pyproject-fmt-rust v1.1.1 (/home/runner/...
Finished release [optimized] target(s) in 4.25s
📦 Including files matching "rust-toolchain.toml"
📦 Built wheel for PyPy 3.8 to dist/pyproject_fmt_rust-...
Compiling pyo3-build-config v0.21.2
Compiling pyo3-macros-backend v0.21.2
Compiling pyo3-ffi v0.21.2
Compiling pyo3 v0.21.2
Compiling pyo3-macros v0.21.2
Compiling pyproject-fmt-rust v1.1.1 (/home/runner/...
Finished release [optimized] target(s) in 4.24s
...
33. Resources
Scienti
fi
c-python/cookie
33
11 b
a
ckends
Cookiecutter & copier supported
In sync with the dev guide
Gener
a
tion tested by nox
🎤 The name of your project
myprog
🎤 The name of your (GitHub?) org
henryiii
🎤 The url to your GitHub or GitLab repository
https://github.com/henryiii/myprog
🎤 Your name
Henry Schreiner
🎤 Your email
a@b.com
🎤 A short description of your project
A great package.
🎤 Select a license
BSD
🎤 Choose a build backend
Scikit-build-core - Compiled C++ (recommended)
🎤 Use version control for versioning
No
Copying from template version 2024.4.23
create .
create CMakeLists.txt
create .pre-commit-config.yaml
create pyproject.toml
create tests
create tests/test_package.py
create tests/test_compiled.py
create .git_archival.txt
create LICENSE
create docs
create docs/conf.py
create docs/index.md
create README.md
create .gitignore
create .github
create .github/workflows
create .github/workflows/cd.yml
create .github/workflows/ci.yml
create .github/CONTRIBUTING.md
create .github/dependabot.yml
create .gitattributes
create .copier-answers.yml
create noxfile.py
create .readthedocs.yaml
create src
create src/main.cpp
create src/myprog
create src/myprog/__init__.py
create src/myprog/_core.pyi
create src/myprog/py.typed
34. Resources
Repo-review
34
Python 3.10+ fr
a
mework
sp-repo-review: checks from guide
Supports WebAssembly
a
nd CLI
(
a
nd pre-commit, etc)
A checker for your checker con
f
ig!
Checks linked to integr
a
ted t
a
gs in guide
V
a
lid
a
te-pyproject
a
lso integr
a
ted
35. Summary
So much has happened in the last two years!
• Three f
a
nt
a
stic build-b
a
ckends
a
re now
a
v
a
il
a
ble for bin
a
ry projects
• Gre
a
t resources
a
v
a
il
a
ble for developing p
a
ck
a
ges
• You c
a
n ship C/C++/Rust code for
a
ll m
a
jor pl
a
tforms even for
a
sm
a
ll project
35
36. Credits and thanks
36
scikit-build te
a
m
Je
a
n-Christophe Fillion-Robin (Kitw
a
re)
M
a
tt McCormick (Kitw
a
re)
Cristi
a
n Le
meson-python te
a
m
R
a
lf Gommers
D
a
niele Nicolodi
Henry Schreiner
Thom
a
s Li
Filipe L
a
íns (Emeritus)
cibuildwheel te
a
m
Joe Rickerby
Y
a
nnick J
a
doul
M
a
tthieu D
a
rbois
Grzegorz Bokot
a
And th
a
nks to the m
a
turin developers for th
a
t f
a
nt
a
stic tool!
Thanks to my other team members!