SlideShare a Scribd company logo
1 of 173
Download to read offline
pip internals
Xavier Fernandez - @xavierfernandez
Polyconseil
July 20, 2015
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 1 / 38
Disclaimer
Disclaimer about this talk
pip only has one ”API”: its Command Line Interface
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 2 / 38
Disclaimer
Disclaimer about this talk
pip only has one ”API”: its Command Line Interface
so internals are subjects to changes
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 2 / 38
Disclaimer
Disclaimer about this talk
pip only has one ”API”: its Command Line Interface
so internals are subjects to changes
corresponds to pip 7.1.0 code
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 2 / 38
Outline for section 1
1 How does pip resolve dependencies ?
2 How does pip perform an installation ?
3 How does pip select the packages ?
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 3 / 38
1 - How does pip resolve dependencies ?
pip.req set.RequirementSet
direct args (my package, https://foo.com/bar-1.0.0.tar.gz,
./my package)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
1 - How does pip resolve dependencies ?
pip.req set.RequirementSet
direct args (my package, https://foo.com/bar-1.0.0.tar.gz,
./my package)
-e/--editable editable package
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
1 - How does pip resolve dependencies ?
pip.req set.RequirementSet
direct args (my package, https://foo.com/bar-1.0.0.tar.gz,
./my package)
-e/--editable editable package
-r/--requirement requirements.txt
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
1 - How does pip resolve dependencies ?
pip.req set.RequirementSet
direct args (my package, https://foo.com/bar-1.0.0.tar.gz,
./my package)
-e/--editable editable package
-r/--requirement requirements.txt
-c/--constraint constraints.txt (since 7.1)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
1 - How does pip resolve dependencies ?
pip.req set.RequirementSet
direct args (my package, https://foo.com/bar-1.0.0.tar.gz,
./my package)
-e/--editable editable package
-r/--requirement requirements.txt
-c/--constraint constraints.txt (since 7.1)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
1 - How does pip resolve dependencies ?
pip.req set.RequirementSet
direct args (my package, https://foo.com/bar-1.0.0.tar.gz,
./my package)
-e/--editable editable package
-r/--requirement requirements.txt
-c/--constraint constraints.txt (since 7.1)
Or a combination
pip install -e ./pkg1 django -e ./pkg2 -r reqs.txt -c
constraint.txt
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
1 - How does pip resolve dependencies ?
RequirementCommand.populate requirement set
from command options
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 5 / 38
1 - How does pip resolve dependencies ?
RequirementCommand.populate requirement set
from command options
calls RequirementSet.add requirement
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 5 / 38
1 - How does pip resolve dependencies ?
RequirementCommand.populate requirement set
from command options
calls RequirementSet.add requirement
constraints, args, editables, requirement files
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 5 / 38
1 - How does pip resolve dependencies ?
RequirementCommand.populate requirement set
from command options
calls RequirementSet.add requirement
constraints, args, editables, requirement files
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 5 / 38
1 - How does pip resolve dependencies ?
RequirementCommand.populate requirement set
from command options
calls RequirementSet.add requirement
constraints, args, editables, requirement files
RequirementSet.add requirement
checks environment markers (e.g. python version==’2.6’, etc)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 5 / 38
1 - How does pip resolve dependencies ?
Environment markers
defined in PEP345 and extended in PEP426 (draft)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 6 / 38
1 - How does pip resolve dependencies ?
Environment markers
defined in PEP345 and extended in PEP426 (draft)
allows to specify required dependencies on specific environment
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 6 / 38
1 - How does pip resolve dependencies ?
Environment markers
defined in PEP345 and extended in PEP426 (draft)
allows to specify required dependencies on specific environment
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 6 / 38
1 - How does pip resolve dependencies ?
Environment markers
defined in PEP345 and extended in PEP426 (draft)
allows to specify required dependencies on specific environment
Examples
extras require={’:python version=="3.3"’: [’asyncio’]}
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 6 / 38
1 - How does pip resolve dependencies ?
Environment markers
defined in PEP345 and extended in PEP426 (draft)
allows to specify required dependencies on specific environment
Examples
extras require={’:python version=="3.3"’: [’asyncio’]}
extras require={ ’signatures’: [’keyring’],
’signatures:sys platform!="win32"’: [’pyxdg’], }
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 6 / 38
1 - How does pip resolve dependencies ?
RequirementCommand.populate requirement set
from command options
calls RequirementSet.add requirement
constraints, args, editables, requirement files
RequirementSet.add requirement
checks environment markers (e.g. python version==’2.6’, etc)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 7 / 38
1 - How does pip resolve dependencies ?
RequirementCommand.populate requirement set
from command options
calls RequirementSet.add requirement
constraints, args, editables, requirement files
RequirementSet.add requirement
checks environment markers (e.g. python version==’2.6’, etc)
if already there and user supplied, reject any double requirement
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 7 / 38
1 - How does pip resolve dependencies ?
RequirementCommand.populate requirement set
from command options
calls RequirementSet.add requirement
constraints, args, editables, requirement files
RequirementSet.add requirement
checks environment markers (e.g. python version==’2.6’, etc)
if already there and user supplied, reject any double requirement
except if previous one is a constraint, then use constraint version
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 7 / 38
1 - How does pip resolve dependencies ?
Constraint
$ echo "django==1.8.3" > constraint.txt
$ pip install django==1.4.3 -c constraint.txt
Successfully installed django-1.8.3
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 8 / 38
1 - How does pip resolve dependencies ?
RequirementCommand.populate requirement set
from command options
calls RequirementSet.add requirement
constraints, args, editables, requirement files
RequirementSet.add requirement
checks environment markers (e.g. python version==’2.6’, etc)
if already there and user supplied, reject any double requirement
except if previous one is a constraint, then use constraint version
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 9 / 38
1 - How does pip resolve dependencies ?
RequirementCommand.populate requirement set
from command options
calls RequirementSet.add requirement
constraints, args, editables, requirement files
RequirementSet.add requirement
checks environment markers (e.g. python version==’2.6’, etc)
if already there and user supplied, reject any double requirement
except if previous one is a constraint, then use constraint version
if already there and comes from dependency, ignore
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 9 / 38
1 - How does pip resolve dependencies ?
Conflict between user and dependency
package simple with install requires=[’pep8==1.2’]
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 10 / 38
1 - How does pip resolve dependencies ?
Conflict between user and dependency
package simple with install requires=[’pep8==1.2’]
pip install simple pep8==1.3
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 10 / 38
1 - How does pip resolve dependencies ?
Conflict between user and dependency
package simple with install requires=[’pep8==1.2’]
pip install simple pep8==1.3
add requirement(’simple’) and add requirement(pep8==1.3)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 10 / 38
1 - How does pip resolve dependencies ?
Conflict between user and dependency
package simple with install requires=[’pep8==1.2’]
pip install simple pep8==1.3
add requirement(’simple’) and add requirement(pep8==1.3)
add requirement(pep8==1.2) ignored
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 10 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
calls RequirementSet. prepare file
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
calls RequirementSet. prepare file
on unnamed requirements
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
calls RequirementSet. prepare file
on unnamed requirements
named requirements
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
calls RequirementSet. prepare file
on unnamed requirements
named requirements
newly found dependencies
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
calls RequirementSet. prepare file
on unnamed requirements
named requirements
newly found dependencies
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
calls RequirementSet. prepare file
on unnamed requirements
named requirements
newly found dependencies
RequirementSet. prepare file
check if already installed and if new version needed
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
calls RequirementSet. prepare file
on unnamed requirements
named requirements
newly found dependencies
RequirementSet. prepare file
check if already installed and if new version needed
for non-link reqs, use finder.find requirement to populate req.link
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
calls RequirementSet. prepare file
on unnamed requirements
named requirements
newly found dependencies
RequirementSet. prepare file
check if already installed and if new version needed
for non-link reqs, use finder.find requirement to populate req.link
calls req.link.setter and search for a cached wheel for this link
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
calls RequirementSet. prepare file
on unnamed requirements
named requirements
newly found dependencies
RequirementSet. prepare file
check if already installed and if new version needed
for non-link reqs, use finder.find requirement to populate req.link
calls req.link.setter and search for a cached wheel for this link
download/update/unpack source files
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
calls RequirementSet. prepare file
on unnamed requirements
named requirements
newly found dependencies
RequirementSet. prepare file
check if already installed and if new version needed
for non-link reqs, use finder.find requirement to populate req.link
calls req.link.setter and search for a cached wheel for this link
download/update/unpack source files
compute metadata: setup.py egg info or directly from wheel
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
calls RequirementSet. prepare file
on unnamed requirements
named requirements
newly found dependencies
RequirementSet. prepare file
check if already installed and if new version needed
for non-link reqs, use finder.find requirement to populate req.link
calls req.link.setter and search for a cached wheel for this link
download/update/unpack source files
compute metadata: setup.py egg info or directly from wheel
compute dependencies (with extras)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
pip install flake8
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 12 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
pip install flake8
adds pep8, pyflakes, mccabe
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 12 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
pip install flake8
adds pep8, pyflakes, mccabe
populate links for all
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 12 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
pip install flake8
adds pep8, pyflakes, mccabe
populate links for all
at the end, in a temporary directory (build location)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 12 / 38
1 - How does pip resolve dependencies ?
RequirementSet.prepare files
pip install flake8
adds pep8, pyflakes, mccabe
populate links for all
at the end, in a temporary directory (build location)
4 directories, one for each dependency
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 12 / 38
1 - How does pip resolve dependencies ?
Optional Wheel caching (pip 7+)
if wheel package available
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
1 - How does pip resolve dependencies ?
Optional Wheel caching (pip 7+)
if wheel package available
if not using options --download or --no-cache-dir
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
1 - How does pip resolve dependencies ?
Optional Wheel caching (pip 7+)
if wheel package available
if not using options --download or --no-cache-dir
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
1 - How does pip resolve dependencies ?
Optional Wheel caching (pip 7+)
if wheel package available
if not using options --download or --no-cache-dir
Exceptions
already wheels
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
1 - How does pip resolve dependencies ?
Optional Wheel caching (pip 7+)
if wheel package available
if not using options --download or --no-cache-dir
Exceptions
already wheels
editable requirements / requirements pointing to a VCS
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
1 - How does pip resolve dependencies ?
Optional Wheel caching (pip 7+)
if wheel package available
if not using options --download or --no-cache-dir
Exceptions
already wheels
editable requirements / requirements pointing to a VCS
--no-binary was used with :all:/the specific package
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
1 - How does pip resolve dependencies ?
Optional Wheel caching (pip 7+)
if wheel package available
if not using options --download or --no-cache-dir
Exceptions
already wheels
editable requirements / requirements pointing to a VCS
--no-binary was used with :all:/the specific package
other special cases
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
1 - How does pip resolve dependencies ?
Optional Wheel caching (pip 7+)
if wheel package available
if not using options --download or --no-cache-dir
Exceptions
already wheels
editable requirements / requirements pointing to a VCS
--no-binary was used with :all:/the specific package
other special cases
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
1 - How does pip resolve dependencies ?
Optional Wheel caching (pip 7+)
if wheel package available
if not using options --download or --no-cache-dir
Exceptions
already wheels
editable requirements / requirements pointing to a VCS
--no-binary was used with :all:/the specific package
other special cases
Wheel caching
on success, update link to point on new wheel file and unpack the
wheel in the build location
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
1 - How does pip resolve dependencies ?
Optional Wheel caching (pip 7+)
if wheel package available
if not using options --download or --no-cache-dir
Exceptions
already wheels
editable requirements / requirements pointing to a VCS
--no-binary was used with :all:/the specific package
other special cases
Wheel caching
on success, update link to point on new wheel file and unpack the
wheel in the build location
else, no issue
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
Outline for section 2
1 How does pip resolve dependencies ?
2 How does pip perform an installation ?
3 How does pip select the packages ?
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 14 / 38
2 - How does pip perform an installation ?
Wheels
already unpacked in prepare files
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
2 - How does pip perform an installation ?
Wheels
already unpacked in prepare files
move directory to site-packages
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
2 - How does pip perform an installation ?
Wheels
already unpacked in prepare files
move directory to site-packages
some complexities for scripts/data
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
2 - How does pip perform an installation ?
Wheels
already unpacked in prepare files
move directory to site-packages
some complexities for scripts/data
RECORD file
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
2 - How does pip perform an installation ?
Wheels
already unpacked in prepare files
move directory to site-packages
some complexities for scripts/data
RECORD file
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
2 - How does pip perform an installation ?
Wheels
already unpacked in prepare files
move directory to site-packages
some complexities for scripts/data
RECORD file
Note
No setuptools needed
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
2 - How does pip perform an installation ?
setup.py install
req install.InstallRequirement.install
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
2 - How does pip perform an installation ?
setup.py install
req install.InstallRequirement.install
import setuptools; exec(compile(open(’setup.py’))
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
2 - How does pip perform an installation ?
setup.py install
req install.InstallRequirement.install
import setuptools; exec(compile(open(’setup.py’))
python setup.py install with some options, mainly:
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
2 - How does pip perform an installation ?
setup.py install
req install.InstallRequirement.install
import setuptools; exec(compile(open(’setup.py’))
python setup.py install with some options, mainly:
--record tmp path/installed-files.txt
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
2 - How does pip perform an installation ?
setup.py install
req install.InstallRequirement.install
import setuptools; exec(compile(open(’setup.py’))
python setup.py install with some options, mainly:
--record tmp path/installed-files.txt
--single-version-externally-managed
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
2 - How does pip perform an installation ?
setup.py install
req install.InstallRequirement.install
import setuptools; exec(compile(open(’setup.py’))
python setup.py install with some options, mainly:
--record tmp path/installed-files.txt
--single-version-externally-managed
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
2 - How does pip perform an installation ?
setup.py install
req install.InstallRequirement.install
import setuptools; exec(compile(open(’setup.py’))
python setup.py install with some options, mainly:
--record tmp path/installed-files.txt
--single-version-externally-managed
Avoid python setup.py install
with setuptools: easy install invocation
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
2 - How does pip perform an installation ?
setup.py install
req install.InstallRequirement.install
import setuptools; exec(compile(open(’setup.py’))
python setup.py install with some options, mainly:
--record tmp path/installed-files.txt
--single-version-externally-managed
Avoid python setup.py install
with setuptools: easy install invocation
with distutils: no installed-files.txt
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
2 - How does pip perform an installation ?
setup.py install
req install.InstallRequirement.install
import setuptools; exec(compile(open(’setup.py’))
python setup.py install with some options, mainly:
--record tmp path/installed-files.txt
--single-version-externally-managed
Avoid python setup.py install
with setuptools: easy install invocation
with distutils: no installed-files.txt
prefer pip install .
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
2 - How does pip perform an installation ?
editable
req install.InstallRequirement.install editable
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 17 / 38
2 - How does pip perform an installation ?
editable
req install.InstallRequirement.install editable
import setuptools; exec(compile(open(’setup.py’))
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 17 / 38
2 - How does pip perform an installation ?
editable
req install.InstallRequirement.install editable
import setuptools; exec(compile(open(’setup.py’))
python setup.py develop --no-deps
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 17 / 38
2 - How does pip perform an installation ?
Rollbacks
uninstalls old version first
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
2 - How does pip perform an installation ?
Rollbacks
uninstalls old version first
tries to install
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
2 - How does pip perform an installation ?
Rollbacks
uninstalls old version first
tries to install
on success, all good, commits uninstall
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
2 - How does pip perform an installation ?
Rollbacks
uninstalls old version first
tries to install
on success, all good, commits uninstall
else, rollbacks uninstall
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
2 - How does pip perform an installation ?
Rollbacks
uninstalls old version first
tries to install
on success, all good, commits uninstall
else, rollbacks uninstall
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
2 - How does pip perform an installation ?
Rollbacks
uninstalls old version first
tries to install
on success, all good, commits uninstall
else, rollbacks uninstall
Only one rollback
rollbacks concerns the last uninstall
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
2 - How does pip perform an installation ?
Rollbacks
uninstalls old version first
tries to install
on success, all good, commits uninstall
else, rollbacks uninstall
Only one rollback
rollbacks concerns the last uninstall
not the whole install call
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
2 - How does pip perform an installation ?
Rollbacks
A==1.0 requires B==1.0 and A==2.0 requires B==2.0
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
2 - How does pip perform an installation ?
Rollbacks
A==1.0 requires B==1.0 and A==2.0 requires B==2.0
A1 and B1 in your venv
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
2 - How does pip perform an installation ?
Rollbacks
A==1.0 requires B==1.0 and A==2.0 requires B==2.0
A1 and B1 in your venv
pip install A==2.0
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
2 - How does pip perform an installation ?
Rollbacks
A==1.0 requires B==1.0 and A==2.0 requires B==2.0
A1 and B1 in your venv
pip install A==2.0
it might: uninstall B1 and install B2 successfully
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
2 - How does pip perform an installation ?
Rollbacks
A==1.0 requires B==1.0 and A==2.0 requires B==2.0
A1 and B1 in your venv
pip install A==2.0
it might: uninstall B1 and install B2 successfully
but then might uninstall A1 and crash on A2 install
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
2 - How does pip perform an installation ?
Rollbacks
A==1.0 requires B==1.0 and A==2.0 requires B==2.0
A1 and B1 in your venv
pip install A==2.0
it might: uninstall B1 and install B2 successfully
but then might uninstall A1 and crash on A2 install
A1 will be restored
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
2 - How does pip perform an installation ?
Rollbacks
A==1.0 requires B==1.0 and A==2.0 requires B==2.0
A1 and B1 in your venv
pip install A==2.0
it might: uninstall B1 and install B2 successfully
but then might uninstall A1 and crash on A2 install
A1 will be restored
but you end up with A1 and B2
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
2 - How does pip perform an installation ?
Installation order
tries to respect the provided (from args) order
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
2 - How does pip perform an installation ?
Installation order
tries to respect the provided (from args) order
but installs dependencies first
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
2 - How does pip perform an installation ?
Installation order
tries to respect the provided (from args) order
but installs dependencies first
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
2 - How does pip perform an installation ?
Installation order
tries to respect the provided (from args) order
but installs dependencies first
Implementation detail
No order guarantee
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
2 - How does pip perform an installation ?
Installation order
tries to respect the provided (from args) order
but installs dependencies first
Implementation detail
No order guarantee
if you need this order, use setup requires
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
2 - How does pip perform an installation ?
Installation order
tries to respect the provided (from args) order
but installs dependencies first
Implementation detail
No order guarantee
if you need this order, use setup requires
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
2 - How does pip perform an installation ?
Installation order
tries to respect the provided (from args) order
but installs dependencies first
Implementation detail
No order guarantee
if you need this order, use setup requires
setup requires
currently ”owned” by setuptools
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
2 - How does pip perform an installation ?
Installation order
tries to respect the provided (from args) order
but installs dependencies first
Implementation detail
No order guarantee
if you need this order, use setup requires
setup requires
currently ”owned” by setuptools
might call easy install
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
2 - How does pip perform an installation ?
Installation order
tries to respect the provided (from args) order
but installs dependencies first
Implementation detail
No order guarantee
if you need this order, use setup requires
setup requires
currently ”owned” by setuptools
might call easy install
hopefully, pip might take control of this feature
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
Outline for section 3
1 How does pip resolve dependencies ?
2 How does pip perform an installation ?
3 How does pip select the packages ?
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 21 / 38
3 - How does pip select the packages ?
Context
pip install foo bar
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 22 / 38
3 - How does pip select the packages ?
Context
pip install foo bar
https://some pypi.com/simple/foo bar/foo bar-1.2.tar.gz
https://some pypi.com/simple/foo bar/foo bar-1.0-py3-none-any.whl
/home/user/wheelhouse/foo bar-3.0-py3-none-any.whl
dozens of others...
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 22 / 38
3 - How does pip select the packages ?
Context
pip install foo bar
https://some pypi.com/simple/foo bar/foo bar-1.2.tar.gz
https://some pypi.com/simple/foo bar/foo bar-1.0-py3-none-any.whl
/home/user/wheelhouse/foo bar-3.0-py3-none-any.whl
dozens of others...
What file should pip use ?
different versions
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 22 / 38
3 - How does pip select the packages ?
Context
pip install foo bar
https://some pypi.com/simple/foo bar/foo bar-1.2.tar.gz
https://some pypi.com/simple/foo bar/foo bar-1.0-py3-none-any.whl
/home/user/wheelhouse/foo bar-3.0-py3-none-any.whl
dozens of others...
What file should pip use ?
different versions
different formats
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 22 / 38
3 - How does pip select the packages ?
Context
pip install foo bar
https://some pypi.com/simple/foo bar/foo bar-1.2.tar.gz
https://some pypi.com/simple/foo bar/foo bar-1.0-py3-none-any.whl
/home/user/wheelhouse/foo bar-3.0-py3-none-any.whl
dozens of others...
What file should pip use ?
different versions
different formats
different locations
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 22 / 38
3 - How does pip select the packages ?
Where the magic happens: pip/index.py
PackageFinder.find requirement(req, upgrade)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 23 / 38
3 - How does pip select the packages ?
Where the magic happens: pip/index.py
PackageFinder.find requirement(req, upgrade)
req is an InstallRequirement: django, django==1.8.3,
Django<1.8
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 23 / 38
3 - How does pip select the packages ?
Where the magic happens: pip/index.py
PackageFinder.find requirement(req, upgrade)
req is an InstallRequirement: django, django==1.8.3,
Django<1.8
upgrade is a boolean (--upgrade option)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 23 / 38
3 - How does pip select the packages ?
Where the magic happens: pip/index.py
PackageFinder.find requirement(req, upgrade)
req is an InstallRequirement: django, django==1.8.3,
Django<1.8
upgrade is a boolean (--upgrade option)
returns an InstallationCandidate (a file/location)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 23 / 38
3 - How does pip select the packages ?
find requirement(req, upgrade)
calls self. find all versions(req.name)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
3 - How does pip select the packages ?
find requirement(req, upgrade)
calls self. find all versions(req.name)
for django on PyPI, 112 InstallationCandidate
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
3 - How does pip select the packages ?
find requirement(req, upgrade)
calls self. find all versions(req.name)
for django on PyPI, 112 InstallationCandidate
on my setup (with private mirror and wheelhouse), 249
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
3 - How does pip select the packages ?
find requirement(req, upgrade)
calls self. find all versions(req.name)
for django on PyPI, 112 InstallationCandidate
on my setup (with private mirror and wheelhouse), 249
filter on version requirements (==1.8.3) and --pre option
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
3 - How does pip select the packages ?
find requirement(req, upgrade)
calls self. find all versions(req.name)
for django on PyPI, 112 InstallationCandidate
on my setup (with private mirror and wheelhouse), 249
filter on version requirements (==1.8.3) and --pre option
(stable) sort by version
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
3 - How does pip select the packages ?
find requirement(req, upgrade)
calls self. find all versions(req.name)
for django on PyPI, 112 InstallationCandidate
on my setup (with private mirror and wheelhouse), 249
filter on version requirements (==1.8.3) and --pre option
(stable) sort by version
returns most recent/currently installed version (--upgrade option)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
3 - How does pip select the packages ?
find all versions: Three main sources
indexes: --index-url, --extra-index-url or --no-index options
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 25 / 38
3 - How does pip select the packages ?
find all versions: Three main sources
indexes: --index-url, --extra-index-url or --no-index options
find links: --find-links options
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 25 / 38
3 - How does pip select the packages ?
find all versions: Three main sources
indexes: --index-url, --extra-index-url or --no-index options
find links: --find-links options
dependency links: provided by some setup.py if
--process-dependency-links
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 25 / 38
3 - How does pip select the packages ?
Indexes
expects something like scheme://some index/
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
3 - How does pip select the packages ?
Indexes
expects something like scheme://some index/
goes to scheme://some index/package name to list available files
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
3 - How does pip select the packages ?
Indexes
expects something like scheme://some index/
goes to scheme://some index/package name to list available files
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
3 - How does pip select the packages ?
Indexes
expects something like scheme://some index/
goes to scheme://some index/package name to list available files
Normalized name
python-dateutil
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
3 - How does pip select the packages ?
Indexes
expects something like scheme://some index/
goes to scheme://some index/package name to list available files
Normalized name
python-dateutil
Python-Dateutil
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
3 - How does pip select the packages ?
Indexes
expects something like scheme://some index/
goes to scheme://some index/package name to list available files
Normalized name
python-dateutil
Python-Dateutil
python dateutil
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
3 - How does pip select the packages ?
Indexes
expects something like scheme://some index/
goes to scheme://some index/package name to list available files
Normalized name
python-dateutil
Python-Dateutil
python dateutil
pYthOn datEutiL
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
3 - How does pip select the packages ?
With modern indexes - PyPI
GET on https://pypi/simple/Python Dateutil
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
3 - How does pip select the packages ?
With modern indexes - PyPI
GET on https://pypi/simple/Python Dateutil
redirects to https://pypi/simple/python-dateutil
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
3 - How does pip select the packages ?
With modern indexes - PyPI
GET on https://pypi/simple/Python Dateutil
redirects to https://pypi/simple/python-dateutil
re-redirects to https://pypi/simple/python-dateutil/
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
3 - How does pip select the packages ?
With modern indexes - PyPI
GET on https://pypi/simple/Python Dateutil
redirects to https://pypi/simple/python-dateutil
re-redirects to https://pypi/simple/python-dateutil/
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
3 - How does pip select the packages ?
With modern indexes - PyPI
GET on https://pypi/simple/Python Dateutil
redirects to https://pypi/simple/python-dateutil
re-redirects to https://pypi/simple/python-dateutil/
Older/simpler indexes
GET on https://pypi/simple/Python Dateutil returns a 404
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
3 - How does pip select the packages ?
With modern indexes - PyPI
GET on https://pypi/simple/Python Dateutil
redirects to https://pypi/simple/python-dateutil
re-redirects to https://pypi/simple/python-dateutil/
Older/simpler indexes
GET on https://pypi/simple/Python Dateutil returns a 404
try GET on https://pypi/simple/ - (2,7M for PyPI)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
3 - How does pip select the packages ?
With modern indexes - PyPI
GET on https://pypi/simple/Python Dateutil
redirects to https://pypi/simple/python-dateutil
re-redirects to https://pypi/simple/python-dateutil/
Older/simpler indexes
GET on https://pypi/simple/Python Dateutil returns a 404
try GET on https://pypi/simple/ - (2,7M for PyPI)
match all the links of the index on the normalized form
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
3 - How does pip select the packages ?
With modern indexes - PyPI
GET on https://pypi/simple/Python Dateutil
redirects to https://pypi/simple/python-dateutil
re-redirects to https://pypi/simple/python-dateutil/
Older/simpler indexes
GET on https://pypi/simple/Python Dateutil returns a 404
try GET on https://pypi/simple/ - (2,7M for PyPI)
match all the links of the index on the normalized form
Note: this is only performed on the first index but used on all
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
3 - How does pip select the packages ?
With modern indexes - PyPI
GET on https://pypi/simple/Python Dateutil
redirects to https://pypi/simple/python-dateutil
re-redirects to https://pypi/simple/python-dateutil/
Older/simpler indexes
GET on https://pypi/simple/Python Dateutil returns a 404
try GET on https://pypi/simple/ - (2,7M for PyPI)
match all the links of the index on the normalized form
Note: this is only performed on the first index but used on all
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
3 - How does pip select the packages ?
With modern indexes - PyPI
GET on https://pypi/simple/Python Dateutil
redirects to https://pypi/simple/python-dateutil
re-redirects to https://pypi/simple/python-dateutil/
Older/simpler indexes
GET on https://pypi/simple/Python Dateutil returns a 404
try GET on https://pypi/simple/ - (2,7M for PyPI)
match all the links of the index on the normalized form
Note: this is only performed on the first index but used on all
New locations
https://pypi/simple/
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
3 - How does pip select the packages ?
With modern indexes - PyPI
GET on https://pypi/simple/Python Dateutil
redirects to https://pypi/simple/python-dateutil
re-redirects to https://pypi/simple/python-dateutil/
Older/simpler indexes
GET on https://pypi/simple/Python Dateutil returns a 404
try GET on https://pypi/simple/ - (2,7M for PyPI)
match all the links of the index on the normalized form
Note: this is only performed on the first index but used on all
New locations
https://pypi/simple/
https://pypi/simple/python-dateutil
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 28 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
sort locations
file locations url locations
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 28 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
sort locations
file locations url locations
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 28 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
sort locations
file locations url locations
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 28 / 38
3 - How does pip select the packages ?
sort locations
if the scheme == file or if the path exists:
if is a file:
if html: -> urls
else: -> files
elif is a directory:
if dealing with find_links:
for all files:
if html: -> urls
else: -> files
else: -> urls
else: -> urls
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 29 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
sort locations
file locations url locations
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 30 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
sort locations
file locations url locations find links locations
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 30 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
sort locations
file locations url locations find links locations dep links locations
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 30 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
sort locations
file locations url locations find links locations dep links locations
validate secure origin
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 30 / 38
3 - How does pip select the packages ?
validate secure origin for url locations
protocol, hostname, port
(”https”, ”*”, ”*”),
(”*”, ”localhost”, ”*”),
(”*”, ”127.0.0.0/8”, ”*”),
(”*”, ”::1/128”, ”*”),
(”file”, ”*”, None),
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 31 / 38
3 - How does pip select the packages ?
validate secure origin for url locations
protocol, hostname, port
(”https”, ”*”, ”*”),
(”*”, ”localhost”, ”*”),
(”*”, ”127.0.0.0/8”, ”*”),
(”*”, ”::1/128”, ”*”),
(”file”, ”*”, None),
(”*”, host, ”*”), for all hosts from --trusted-host option
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 31 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
sort locations
file locations url locations find links locations dep links locations
validate secure origin
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 32 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
sort locations
file locations url locations find links locations dep links locations
validate secure origin
HTMLPage
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 32 / 38
3 - How does pip select the packages ?
HTMLPage
for urls looking like archives, check content type first
if ”text/html”, continue
get content and parse
check all links
if rel contains ’homepage’ or ’download’
add it (untrusted to url locations)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 33 / 38
3 - How does pip select the packages ?
HTMLPage
for urls looking like archives, check content type first
if ”text/html”, continue
get content and parse
check all links
if rel contains ’homepage’ or ’download’
add it (untrusted to url locations)
Note
with api version 2 i.e. PyPI, <a> can be marked with
rel="internal"
only trust internals
<meta name="api-version" value="2" /> in the page head
--allow-external/--allow-all-external
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 33 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
sort locations
file locations url locations find links locations dep links locations
validate secure origin
HTMLPage
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 34 / 38
3 - How does pip select the packages ?
indexes
index links find links dependency links
find url name
sort locations
file locations url locations find links locations dep links locations
validate secure origin
HTMLPage
package versions
links
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 34 / 38
3 - How does pip select the packages ?
package versions filter
push egg links (containing #egg=) to the end
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
3 - How does pip select the packages ?
package versions filter
push egg links (containing #egg=) to the end
select/reject on lots of criteria
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
3 - How does pip select the packages ?
package versions filter
push egg links (containing #egg=) to the end
select/reject on lots of criteria
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
3 - How does pip select the packages ?
package versions filter
push egg links (containing #egg=) to the end
select/reject on lots of criteria
Criteria
unsupported format (like bz2 on some install)
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
3 - How does pip select the packages ?
package versions filter
push egg links (containing #egg=) to the end
select/reject on lots of criteria
Criteria
unsupported format (like bz2 on some install)
macos10x files ending in .zip
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
3 - How does pip select the packages ?
package versions filter
push egg links (containing #egg=) to the end
select/reject on lots of criteria
Criteria
unsupported format (like bz2 on some install)
macos10x files ending in .zip
--only-binary/--no-binary options
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
3 - How does pip select the packages ?
package versions filter
push egg links (containing #egg=) to the end
select/reject on lots of criteria
Criteria
unsupported format (like bz2 on some install)
macos10x files ending in .zip
--only-binary/--no-binary options
for wheel files: check compatibility and for non-windows/macos
installs, refuse platform specific wheels from PyPI
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
3 - How does pip select the packages ?
package versions filter
push egg links (containing #egg=) to the end
select/reject on lots of criteria
Criteria
unsupported format (like bz2 on some install)
macos10x files ending in .zip
--only-binary/--no-binary options
for wheel files: check compatibility and for non-windows/macos
installs, refuse platform specific wheels from PyPI
tries to extract version from filename and reject file if fails
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
3 - How does pip select the packages ?
package versions filter
push egg links (containing #egg=) to the end
select/reject on lots of criteria
Criteria
unsupported format (like bz2 on some install)
macos10x files ending in .zip
--only-binary/--no-binary options
for wheel files: check compatibility and for non-windows/macos
installs, refuse platform specific wheels from PyPI
tries to extract version from filename and reject file if fails
if version contains -py2/-py3/-py27/etc, check python version
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
3 - How does pip select the packages ?
package versions filter
push egg links (containing #egg=) to the end
select/reject on lots of criteria
Criteria
unsupported format (like bz2 on some install)
macos10x files ending in .zip
--only-binary/--no-binary options
for wheel files: check compatibility and for non-windows/macos
installs, refuse platform specific wheels from PyPI
tries to extract version from filename and reject file if fails
if version contains -py2/-py3/-py27/etc, check python version
check internal/external hosted files and verifiable/unverifiable files
--allow-external/--allow-all-external/--allow-unverified
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
3 - How does pip select the packages ?
Preference Order
file sources are prefered
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
3 - How does pip select the packages ?
Preference Order
file sources are prefered
then direct find links files
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
3 - How does pip select the packages ?
Preference Order
file sources are prefered
then direct find links files
then urls links
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
3 - How does pip select the packages ?
Preference Order
file sources are prefered
then direct find links files
then urls links
direct dependency links files
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
3 - How does pip select the packages ?
Preference Order
file sources are prefered
then direct find links files
then urls links
direct dependency links files
this is what find all versions returns
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
3 - How does pip select the packages ?
Preference Order
file sources are prefered
then direct find links files
then urls links
direct dependency links files
this is what find all versions returns
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
3 - How does pip select the packages ?
Preference Order
file sources are prefered
then direct find links files
then urls links
direct dependency links files
this is what find all versions returns
Note
Wheel cache (pip 7 feature) has no impact on find all versions
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
3 - How does pip select the packages ?
find all versions(’pep8’)
[<InstallationCandidate(’pep8’, <Version(’1.5.7’)>,
<Link file:///home/user/wheelhouse/pep8-1.5.7-py2.py3-none-any.whl>)>,
<InstallationCandidate(’pep8’, <Version(’1.5.0’)>,
<Link file:///home/user/wheelhouse/pep8-1.5.0-py27-none-any.whl>)>,
...
<InstallationCandidate(’pep8’, <Version(’0.3.1’)>,
<Link https://private_pypi/simple/pep8/pep8-0.3.1.tar.gz>)>,
<InstallationCandidate(’pep8’, <Version(’0.4.1’)>,
<Link https://private_pypi/simple/pep8/pep8-0.4.1.tar.gz>)>,
...
<InstallationCandidate(’pep8’, <Version(’1.5.2’)>,
<Link https://pypi.python.org/packages/3.3/p/pep8/
pep8-1.5.2-py2.py3-none-any.whl#md5=a0caba277a2f491b1634246a338a1235>)>,
<InstallationCandidate(’pep8’, <Version(’1.5.3’)>,
<Link https://pypi.python.org/packages/3.3/p/pep8/
pep8-1.5.3-py2.py3-none-any.whl#md5=0654904760aa9a24062bf367f39e873e>)>]
Xavier Fernandez (Polyconseil) pip internals July 20, 2015 37 / 38
Thanks!
Thanks
Any questions ?
Thanks!
Thanks
Any questions ?
Bon app´etit !

More Related Content

Similar to pip internals

Jopr Plugin Development
Jopr Plugin DevelopmentJopr Plugin Development
Jopr Plugin Development
SEA Tecnologia
 
Forti gate troubleshooting_guide_v0.10
Forti gate troubleshooting_guide_v0.10Forti gate troubleshooting_guide_v0.10
Forti gate troubleshooting_guide_v0.10
Phong Nguyễn
 

Similar to pip internals (20)

Python packaging and dependency resolution
Python packaging and dependency resolutionPython packaging and dependency resolution
Python packaging and dependency resolution
 
Python virtualenv & pip in 90 minutes
Python virtualenv & pip in 90 minutesPython virtualenv & pip in 90 minutes
Python virtualenv & pip in 90 minutes
 
TPOT: The data science assistant
TPOT: The data science assistantTPOT: The data science assistant
TPOT: The data science assistant
 
First python project
First python projectFirst python project
First python project
 
Christian Strappazzon - Presentazione Python Milano - Codemotion Milano 2017
Christian Strappazzon - Presentazione Python Milano - Codemotion Milano 2017Christian Strappazzon - Presentazione Python Milano - Codemotion Milano 2017
Christian Strappazzon - Presentazione Python Milano - Codemotion Milano 2017
 
Arbeiten mit distribute, pip und virtualenv
Arbeiten mit distribute, pip und virtualenvArbeiten mit distribute, pip und virtualenv
Arbeiten mit distribute, pip und virtualenv
 
Python packaging: how did we get here, and where are we going?
Python packaging: how did we get here, and where are we going?Python packaging: how did we get here, and where are we going?
Python packaging: how did we get here, and where are we going?
 
Slimfast
SlimfastSlimfast
Slimfast
 
Ruby and Rails Packaging to Production
Ruby and Rails Packaging to ProductionRuby and Rails Packaging to Production
Ruby and Rails Packaging to Production
 
The Butler and The Snake (Europython 2015)
The Butler and The Snake (Europython 2015)The Butler and The Snake (Europython 2015)
The Butler and The Snake (Europython 2015)
 
Introduction to robot framework
Introduction to robot frameworkIntroduction to robot framework
Introduction to robot framework
 
Beyond QA
Beyond QABeyond QA
Beyond QA
 
Container Security Scanning by Timo Pagel
Container Security Scanning by Timo PagelContainer Security Scanning by Timo Pagel
Container Security Scanning by Timo Pagel
 
Container Security Scanning by Timo Pagel
Container Security Scanning by Timo PagelContainer Security Scanning by Timo Pagel
Container Security Scanning by Timo Pagel
 
Jopr Plugin Development
Jopr Plugin DevelopmentJopr Plugin Development
Jopr Plugin Development
 
Forti gate troubleshooting_guide_v0.10
Forti gate troubleshooting_guide_v0.10Forti gate troubleshooting_guide_v0.10
Forti gate troubleshooting_guide_v0.10
 
FASTEN: Scaling static analyses to ecosystem, presented at FOSDEM 2020 in Bru...
FASTEN: Scaling static analyses to ecosystem, presented at FOSDEM 2020 in Bru...FASTEN: Scaling static analyses to ecosystem, presented at FOSDEM 2020 in Bru...
FASTEN: Scaling static analyses to ecosystem, presented at FOSDEM 2020 in Bru...
 
Integration Testing on Steroids: Run Your Tests on the Real Things
Integration Testing on Steroids: Run Your Tests on the Real ThingsIntegration Testing on Steroids: Run Your Tests on the Real Things
Integration Testing on Steroids: Run Your Tests on the Real Things
 
Pipfile, pipenv, pip… what?!
Pipfile, pipenv, pip… what?!Pipfile, pipenv, pip… what?!
Pipfile, pipenv, pip… what?!
 
vvvvReadme
vvvvReadmevvvvReadme
vvvvReadme
 

Recently uploaded

Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
VictoriaMetrics
 

Recently uploaded (20)

WSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AIWSO2CON 2024 Slides - Unlocking Value with AI
WSO2CON 2024 Slides - Unlocking Value with AI
 
WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?
 
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
 
WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...
WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...
WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...
 
WSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go PlatformlessWSO2CON2024 - It's time to go Platformless
WSO2CON2024 - It's time to go Platformless
 
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public AdministrationWSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
WSO2CON 2024 - How CSI Piemonte Is Apifying the Public Administration
 
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Boksburg [(+27832195400*)] 🏥 Women's Abortion Clinic in ...
 
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
WSO2Con2024 - Navigating the Digital Landscape: Transforming Healthcare with ...
 
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
WSO2Con2024 - GitOps in Action: Navigating Application Deployment in the Plat...
 
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
WSO2CON 2024 - WSO2's Digital Transformation Journey with Choreo: A Platforml...
 
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of TransformationWSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
WSO2CON 2024 - Designing Event-Driven Enterprises: Stories of Transformation
 
WSO2Con2024 - Hello Choreo Presentation - Kanchana
WSO2Con2024 - Hello Choreo Presentation - KanchanaWSO2Con2024 - Hello Choreo Presentation - Kanchana
WSO2Con2024 - Hello Choreo Presentation - Kanchana
 
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
WSO2CON 2024 - API Management Usage at La Poste and Its Impact on Business an...
 
WSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaSWSO2CON 2024 Slides - Open Source to SaaS
WSO2CON 2024 Slides - Open Source to SaaS
 
WSO2CON2024 - Why Should You Consider Ballerina for Your Next Integration
WSO2CON2024 - Why Should You Consider Ballerina for Your Next IntegrationWSO2CON2024 - Why Should You Consider Ballerina for Your Next Integration
WSO2CON2024 - Why Should You Consider Ballerina for Your Next Integration
 
WSO2CON 2024 - Software Engineering for Digital Businesses
WSO2CON 2024 - Software Engineering for Digital BusinessesWSO2CON 2024 - Software Engineering for Digital Businesses
WSO2CON 2024 - Software Engineering for Digital Businesses
 
WSO2Con2024 - Low-Code Integration Tooling
WSO2Con2024 - Low-Code Integration ToolingWSO2Con2024 - Low-Code Integration Tooling
WSO2Con2024 - Low-Code Integration Tooling
 
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open SourceWSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source
 
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
Large-scale Logging Made Easy: Meetup at Deutsche Bank 2024
 
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
WSO2CON 2024 - IoT Needs CIAM: The Importance of Centralized IAM in a Growing...
 

pip internals

  • 1. pip internals Xavier Fernandez - @xavierfernandez Polyconseil July 20, 2015 Xavier Fernandez (Polyconseil) pip internals July 20, 2015 1 / 38
  • 2. Disclaimer Disclaimer about this talk pip only has one ”API”: its Command Line Interface Xavier Fernandez (Polyconseil) pip internals July 20, 2015 2 / 38
  • 3. Disclaimer Disclaimer about this talk pip only has one ”API”: its Command Line Interface so internals are subjects to changes Xavier Fernandez (Polyconseil) pip internals July 20, 2015 2 / 38
  • 4. Disclaimer Disclaimer about this talk pip only has one ”API”: its Command Line Interface so internals are subjects to changes corresponds to pip 7.1.0 code Xavier Fernandez (Polyconseil) pip internals July 20, 2015 2 / 38
  • 5. Outline for section 1 1 How does pip resolve dependencies ? 2 How does pip perform an installation ? 3 How does pip select the packages ? Xavier Fernandez (Polyconseil) pip internals July 20, 2015 3 / 38
  • 6. 1 - How does pip resolve dependencies ? pip.req set.RequirementSet direct args (my package, https://foo.com/bar-1.0.0.tar.gz, ./my package) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
  • 7. 1 - How does pip resolve dependencies ? pip.req set.RequirementSet direct args (my package, https://foo.com/bar-1.0.0.tar.gz, ./my package) -e/--editable editable package Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
  • 8. 1 - How does pip resolve dependencies ? pip.req set.RequirementSet direct args (my package, https://foo.com/bar-1.0.0.tar.gz, ./my package) -e/--editable editable package -r/--requirement requirements.txt Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
  • 9. 1 - How does pip resolve dependencies ? pip.req set.RequirementSet direct args (my package, https://foo.com/bar-1.0.0.tar.gz, ./my package) -e/--editable editable package -r/--requirement requirements.txt -c/--constraint constraints.txt (since 7.1) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
  • 10. 1 - How does pip resolve dependencies ? pip.req set.RequirementSet direct args (my package, https://foo.com/bar-1.0.0.tar.gz, ./my package) -e/--editable editable package -r/--requirement requirements.txt -c/--constraint constraints.txt (since 7.1) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
  • 11. 1 - How does pip resolve dependencies ? pip.req set.RequirementSet direct args (my package, https://foo.com/bar-1.0.0.tar.gz, ./my package) -e/--editable editable package -r/--requirement requirements.txt -c/--constraint constraints.txt (since 7.1) Or a combination pip install -e ./pkg1 django -e ./pkg2 -r reqs.txt -c constraint.txt Xavier Fernandez (Polyconseil) pip internals July 20, 2015 4 / 38
  • 12. 1 - How does pip resolve dependencies ? RequirementCommand.populate requirement set from command options Xavier Fernandez (Polyconseil) pip internals July 20, 2015 5 / 38
  • 13. 1 - How does pip resolve dependencies ? RequirementCommand.populate requirement set from command options calls RequirementSet.add requirement Xavier Fernandez (Polyconseil) pip internals July 20, 2015 5 / 38
  • 14. 1 - How does pip resolve dependencies ? RequirementCommand.populate requirement set from command options calls RequirementSet.add requirement constraints, args, editables, requirement files Xavier Fernandez (Polyconseil) pip internals July 20, 2015 5 / 38
  • 15. 1 - How does pip resolve dependencies ? RequirementCommand.populate requirement set from command options calls RequirementSet.add requirement constraints, args, editables, requirement files Xavier Fernandez (Polyconseil) pip internals July 20, 2015 5 / 38
  • 16. 1 - How does pip resolve dependencies ? RequirementCommand.populate requirement set from command options calls RequirementSet.add requirement constraints, args, editables, requirement files RequirementSet.add requirement checks environment markers (e.g. python version==’2.6’, etc) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 5 / 38
  • 17. 1 - How does pip resolve dependencies ? Environment markers defined in PEP345 and extended in PEP426 (draft) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 6 / 38
  • 18. 1 - How does pip resolve dependencies ? Environment markers defined in PEP345 and extended in PEP426 (draft) allows to specify required dependencies on specific environment Xavier Fernandez (Polyconseil) pip internals July 20, 2015 6 / 38
  • 19. 1 - How does pip resolve dependencies ? Environment markers defined in PEP345 and extended in PEP426 (draft) allows to specify required dependencies on specific environment Xavier Fernandez (Polyconseil) pip internals July 20, 2015 6 / 38
  • 20. 1 - How does pip resolve dependencies ? Environment markers defined in PEP345 and extended in PEP426 (draft) allows to specify required dependencies on specific environment Examples extras require={’:python version=="3.3"’: [’asyncio’]} Xavier Fernandez (Polyconseil) pip internals July 20, 2015 6 / 38
  • 21. 1 - How does pip resolve dependencies ? Environment markers defined in PEP345 and extended in PEP426 (draft) allows to specify required dependencies on specific environment Examples extras require={’:python version=="3.3"’: [’asyncio’]} extras require={ ’signatures’: [’keyring’], ’signatures:sys platform!="win32"’: [’pyxdg’], } Xavier Fernandez (Polyconseil) pip internals July 20, 2015 6 / 38
  • 22. 1 - How does pip resolve dependencies ? RequirementCommand.populate requirement set from command options calls RequirementSet.add requirement constraints, args, editables, requirement files RequirementSet.add requirement checks environment markers (e.g. python version==’2.6’, etc) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 7 / 38
  • 23. 1 - How does pip resolve dependencies ? RequirementCommand.populate requirement set from command options calls RequirementSet.add requirement constraints, args, editables, requirement files RequirementSet.add requirement checks environment markers (e.g. python version==’2.6’, etc) if already there and user supplied, reject any double requirement Xavier Fernandez (Polyconseil) pip internals July 20, 2015 7 / 38
  • 24. 1 - How does pip resolve dependencies ? RequirementCommand.populate requirement set from command options calls RequirementSet.add requirement constraints, args, editables, requirement files RequirementSet.add requirement checks environment markers (e.g. python version==’2.6’, etc) if already there and user supplied, reject any double requirement except if previous one is a constraint, then use constraint version Xavier Fernandez (Polyconseil) pip internals July 20, 2015 7 / 38
  • 25. 1 - How does pip resolve dependencies ? Constraint $ echo "django==1.8.3" > constraint.txt $ pip install django==1.4.3 -c constraint.txt Successfully installed django-1.8.3 Xavier Fernandez (Polyconseil) pip internals July 20, 2015 8 / 38
  • 26. 1 - How does pip resolve dependencies ? RequirementCommand.populate requirement set from command options calls RequirementSet.add requirement constraints, args, editables, requirement files RequirementSet.add requirement checks environment markers (e.g. python version==’2.6’, etc) if already there and user supplied, reject any double requirement except if previous one is a constraint, then use constraint version Xavier Fernandez (Polyconseil) pip internals July 20, 2015 9 / 38
  • 27. 1 - How does pip resolve dependencies ? RequirementCommand.populate requirement set from command options calls RequirementSet.add requirement constraints, args, editables, requirement files RequirementSet.add requirement checks environment markers (e.g. python version==’2.6’, etc) if already there and user supplied, reject any double requirement except if previous one is a constraint, then use constraint version if already there and comes from dependency, ignore Xavier Fernandez (Polyconseil) pip internals July 20, 2015 9 / 38
  • 28. 1 - How does pip resolve dependencies ? Conflict between user and dependency package simple with install requires=[’pep8==1.2’] Xavier Fernandez (Polyconseil) pip internals July 20, 2015 10 / 38
  • 29. 1 - How does pip resolve dependencies ? Conflict between user and dependency package simple with install requires=[’pep8==1.2’] pip install simple pep8==1.3 Xavier Fernandez (Polyconseil) pip internals July 20, 2015 10 / 38
  • 30. 1 - How does pip resolve dependencies ? Conflict between user and dependency package simple with install requires=[’pep8==1.2’] pip install simple pep8==1.3 add requirement(’simple’) and add requirement(pep8==1.3) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 10 / 38
  • 31. 1 - How does pip resolve dependencies ? Conflict between user and dependency package simple with install requires=[’pep8==1.2’] pip install simple pep8==1.3 add requirement(’simple’) and add requirement(pep8==1.3) add requirement(pep8==1.2) ignored Xavier Fernandez (Polyconseil) pip internals July 20, 2015 10 / 38
  • 32. 1 - How does pip resolve dependencies ? RequirementSet.prepare files calls RequirementSet. prepare file Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
  • 33. 1 - How does pip resolve dependencies ? RequirementSet.prepare files calls RequirementSet. prepare file on unnamed requirements Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
  • 34. 1 - How does pip resolve dependencies ? RequirementSet.prepare files calls RequirementSet. prepare file on unnamed requirements named requirements Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
  • 35. 1 - How does pip resolve dependencies ? RequirementSet.prepare files calls RequirementSet. prepare file on unnamed requirements named requirements newly found dependencies Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
  • 36. 1 - How does pip resolve dependencies ? RequirementSet.prepare files calls RequirementSet. prepare file on unnamed requirements named requirements newly found dependencies Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
  • 37. 1 - How does pip resolve dependencies ? RequirementSet.prepare files calls RequirementSet. prepare file on unnamed requirements named requirements newly found dependencies RequirementSet. prepare file check if already installed and if new version needed Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
  • 38. 1 - How does pip resolve dependencies ? RequirementSet.prepare files calls RequirementSet. prepare file on unnamed requirements named requirements newly found dependencies RequirementSet. prepare file check if already installed and if new version needed for non-link reqs, use finder.find requirement to populate req.link Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
  • 39. 1 - How does pip resolve dependencies ? RequirementSet.prepare files calls RequirementSet. prepare file on unnamed requirements named requirements newly found dependencies RequirementSet. prepare file check if already installed and if new version needed for non-link reqs, use finder.find requirement to populate req.link calls req.link.setter and search for a cached wheel for this link Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
  • 40. 1 - How does pip resolve dependencies ? RequirementSet.prepare files calls RequirementSet. prepare file on unnamed requirements named requirements newly found dependencies RequirementSet. prepare file check if already installed and if new version needed for non-link reqs, use finder.find requirement to populate req.link calls req.link.setter and search for a cached wheel for this link download/update/unpack source files Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
  • 41. 1 - How does pip resolve dependencies ? RequirementSet.prepare files calls RequirementSet. prepare file on unnamed requirements named requirements newly found dependencies RequirementSet. prepare file check if already installed and if new version needed for non-link reqs, use finder.find requirement to populate req.link calls req.link.setter and search for a cached wheel for this link download/update/unpack source files compute metadata: setup.py egg info or directly from wheel Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
  • 42. 1 - How does pip resolve dependencies ? RequirementSet.prepare files calls RequirementSet. prepare file on unnamed requirements named requirements newly found dependencies RequirementSet. prepare file check if already installed and if new version needed for non-link reqs, use finder.find requirement to populate req.link calls req.link.setter and search for a cached wheel for this link download/update/unpack source files compute metadata: setup.py egg info or directly from wheel compute dependencies (with extras) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 11 / 38
  • 43. 1 - How does pip resolve dependencies ? RequirementSet.prepare files pip install flake8 Xavier Fernandez (Polyconseil) pip internals July 20, 2015 12 / 38
  • 44. 1 - How does pip resolve dependencies ? RequirementSet.prepare files pip install flake8 adds pep8, pyflakes, mccabe Xavier Fernandez (Polyconseil) pip internals July 20, 2015 12 / 38
  • 45. 1 - How does pip resolve dependencies ? RequirementSet.prepare files pip install flake8 adds pep8, pyflakes, mccabe populate links for all Xavier Fernandez (Polyconseil) pip internals July 20, 2015 12 / 38
  • 46. 1 - How does pip resolve dependencies ? RequirementSet.prepare files pip install flake8 adds pep8, pyflakes, mccabe populate links for all at the end, in a temporary directory (build location) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 12 / 38
  • 47. 1 - How does pip resolve dependencies ? RequirementSet.prepare files pip install flake8 adds pep8, pyflakes, mccabe populate links for all at the end, in a temporary directory (build location) 4 directories, one for each dependency Xavier Fernandez (Polyconseil) pip internals July 20, 2015 12 / 38
  • 48. 1 - How does pip resolve dependencies ? Optional Wheel caching (pip 7+) if wheel package available Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
  • 49. 1 - How does pip resolve dependencies ? Optional Wheel caching (pip 7+) if wheel package available if not using options --download or --no-cache-dir Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
  • 50. 1 - How does pip resolve dependencies ? Optional Wheel caching (pip 7+) if wheel package available if not using options --download or --no-cache-dir Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
  • 51. 1 - How does pip resolve dependencies ? Optional Wheel caching (pip 7+) if wheel package available if not using options --download or --no-cache-dir Exceptions already wheels Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
  • 52. 1 - How does pip resolve dependencies ? Optional Wheel caching (pip 7+) if wheel package available if not using options --download or --no-cache-dir Exceptions already wheels editable requirements / requirements pointing to a VCS Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
  • 53. 1 - How does pip resolve dependencies ? Optional Wheel caching (pip 7+) if wheel package available if not using options --download or --no-cache-dir Exceptions already wheels editable requirements / requirements pointing to a VCS --no-binary was used with :all:/the specific package Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
  • 54. 1 - How does pip resolve dependencies ? Optional Wheel caching (pip 7+) if wheel package available if not using options --download or --no-cache-dir Exceptions already wheels editable requirements / requirements pointing to a VCS --no-binary was used with :all:/the specific package other special cases Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
  • 55. 1 - How does pip resolve dependencies ? Optional Wheel caching (pip 7+) if wheel package available if not using options --download or --no-cache-dir Exceptions already wheels editable requirements / requirements pointing to a VCS --no-binary was used with :all:/the specific package other special cases Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
  • 56. 1 - How does pip resolve dependencies ? Optional Wheel caching (pip 7+) if wheel package available if not using options --download or --no-cache-dir Exceptions already wheels editable requirements / requirements pointing to a VCS --no-binary was used with :all:/the specific package other special cases Wheel caching on success, update link to point on new wheel file and unpack the wheel in the build location Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
  • 57. 1 - How does pip resolve dependencies ? Optional Wheel caching (pip 7+) if wheel package available if not using options --download or --no-cache-dir Exceptions already wheels editable requirements / requirements pointing to a VCS --no-binary was used with :all:/the specific package other special cases Wheel caching on success, update link to point on new wheel file and unpack the wheel in the build location else, no issue Xavier Fernandez (Polyconseil) pip internals July 20, 2015 13 / 38
  • 58. Outline for section 2 1 How does pip resolve dependencies ? 2 How does pip perform an installation ? 3 How does pip select the packages ? Xavier Fernandez (Polyconseil) pip internals July 20, 2015 14 / 38
  • 59. 2 - How does pip perform an installation ? Wheels already unpacked in prepare files Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
  • 60. 2 - How does pip perform an installation ? Wheels already unpacked in prepare files move directory to site-packages Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
  • 61. 2 - How does pip perform an installation ? Wheels already unpacked in prepare files move directory to site-packages some complexities for scripts/data Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
  • 62. 2 - How does pip perform an installation ? Wheels already unpacked in prepare files move directory to site-packages some complexities for scripts/data RECORD file Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
  • 63. 2 - How does pip perform an installation ? Wheels already unpacked in prepare files move directory to site-packages some complexities for scripts/data RECORD file Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
  • 64. 2 - How does pip perform an installation ? Wheels already unpacked in prepare files move directory to site-packages some complexities for scripts/data RECORD file Note No setuptools needed Xavier Fernandez (Polyconseil) pip internals July 20, 2015 15 / 38
  • 65. 2 - How does pip perform an installation ? setup.py install req install.InstallRequirement.install Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
  • 66. 2 - How does pip perform an installation ? setup.py install req install.InstallRequirement.install import setuptools; exec(compile(open(’setup.py’)) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
  • 67. 2 - How does pip perform an installation ? setup.py install req install.InstallRequirement.install import setuptools; exec(compile(open(’setup.py’)) python setup.py install with some options, mainly: Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
  • 68. 2 - How does pip perform an installation ? setup.py install req install.InstallRequirement.install import setuptools; exec(compile(open(’setup.py’)) python setup.py install with some options, mainly: --record tmp path/installed-files.txt Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
  • 69. 2 - How does pip perform an installation ? setup.py install req install.InstallRequirement.install import setuptools; exec(compile(open(’setup.py’)) python setup.py install with some options, mainly: --record tmp path/installed-files.txt --single-version-externally-managed Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
  • 70. 2 - How does pip perform an installation ? setup.py install req install.InstallRequirement.install import setuptools; exec(compile(open(’setup.py’)) python setup.py install with some options, mainly: --record tmp path/installed-files.txt --single-version-externally-managed Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
  • 71. 2 - How does pip perform an installation ? setup.py install req install.InstallRequirement.install import setuptools; exec(compile(open(’setup.py’)) python setup.py install with some options, mainly: --record tmp path/installed-files.txt --single-version-externally-managed Avoid python setup.py install with setuptools: easy install invocation Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
  • 72. 2 - How does pip perform an installation ? setup.py install req install.InstallRequirement.install import setuptools; exec(compile(open(’setup.py’)) python setup.py install with some options, mainly: --record tmp path/installed-files.txt --single-version-externally-managed Avoid python setup.py install with setuptools: easy install invocation with distutils: no installed-files.txt Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
  • 73. 2 - How does pip perform an installation ? setup.py install req install.InstallRequirement.install import setuptools; exec(compile(open(’setup.py’)) python setup.py install with some options, mainly: --record tmp path/installed-files.txt --single-version-externally-managed Avoid python setup.py install with setuptools: easy install invocation with distutils: no installed-files.txt prefer pip install . Xavier Fernandez (Polyconseil) pip internals July 20, 2015 16 / 38
  • 74. 2 - How does pip perform an installation ? editable req install.InstallRequirement.install editable Xavier Fernandez (Polyconseil) pip internals July 20, 2015 17 / 38
  • 75. 2 - How does pip perform an installation ? editable req install.InstallRequirement.install editable import setuptools; exec(compile(open(’setup.py’)) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 17 / 38
  • 76. 2 - How does pip perform an installation ? editable req install.InstallRequirement.install editable import setuptools; exec(compile(open(’setup.py’)) python setup.py develop --no-deps Xavier Fernandez (Polyconseil) pip internals July 20, 2015 17 / 38
  • 77. 2 - How does pip perform an installation ? Rollbacks uninstalls old version first Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
  • 78. 2 - How does pip perform an installation ? Rollbacks uninstalls old version first tries to install Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
  • 79. 2 - How does pip perform an installation ? Rollbacks uninstalls old version first tries to install on success, all good, commits uninstall Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
  • 80. 2 - How does pip perform an installation ? Rollbacks uninstalls old version first tries to install on success, all good, commits uninstall else, rollbacks uninstall Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
  • 81. 2 - How does pip perform an installation ? Rollbacks uninstalls old version first tries to install on success, all good, commits uninstall else, rollbacks uninstall Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
  • 82. 2 - How does pip perform an installation ? Rollbacks uninstalls old version first tries to install on success, all good, commits uninstall else, rollbacks uninstall Only one rollback rollbacks concerns the last uninstall Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
  • 83. 2 - How does pip perform an installation ? Rollbacks uninstalls old version first tries to install on success, all good, commits uninstall else, rollbacks uninstall Only one rollback rollbacks concerns the last uninstall not the whole install call Xavier Fernandez (Polyconseil) pip internals July 20, 2015 18 / 38
  • 84. 2 - How does pip perform an installation ? Rollbacks A==1.0 requires B==1.0 and A==2.0 requires B==2.0 Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
  • 85. 2 - How does pip perform an installation ? Rollbacks A==1.0 requires B==1.0 and A==2.0 requires B==2.0 A1 and B1 in your venv Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
  • 86. 2 - How does pip perform an installation ? Rollbacks A==1.0 requires B==1.0 and A==2.0 requires B==2.0 A1 and B1 in your venv pip install A==2.0 Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
  • 87. 2 - How does pip perform an installation ? Rollbacks A==1.0 requires B==1.0 and A==2.0 requires B==2.0 A1 and B1 in your venv pip install A==2.0 it might: uninstall B1 and install B2 successfully Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
  • 88. 2 - How does pip perform an installation ? Rollbacks A==1.0 requires B==1.0 and A==2.0 requires B==2.0 A1 and B1 in your venv pip install A==2.0 it might: uninstall B1 and install B2 successfully but then might uninstall A1 and crash on A2 install Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
  • 89. 2 - How does pip perform an installation ? Rollbacks A==1.0 requires B==1.0 and A==2.0 requires B==2.0 A1 and B1 in your venv pip install A==2.0 it might: uninstall B1 and install B2 successfully but then might uninstall A1 and crash on A2 install A1 will be restored Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
  • 90. 2 - How does pip perform an installation ? Rollbacks A==1.0 requires B==1.0 and A==2.0 requires B==2.0 A1 and B1 in your venv pip install A==2.0 it might: uninstall B1 and install B2 successfully but then might uninstall A1 and crash on A2 install A1 will be restored but you end up with A1 and B2 Xavier Fernandez (Polyconseil) pip internals July 20, 2015 19 / 38
  • 91. 2 - How does pip perform an installation ? Installation order tries to respect the provided (from args) order Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
  • 92. 2 - How does pip perform an installation ? Installation order tries to respect the provided (from args) order but installs dependencies first Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
  • 93. 2 - How does pip perform an installation ? Installation order tries to respect the provided (from args) order but installs dependencies first Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
  • 94. 2 - How does pip perform an installation ? Installation order tries to respect the provided (from args) order but installs dependencies first Implementation detail No order guarantee Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
  • 95. 2 - How does pip perform an installation ? Installation order tries to respect the provided (from args) order but installs dependencies first Implementation detail No order guarantee if you need this order, use setup requires Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
  • 96. 2 - How does pip perform an installation ? Installation order tries to respect the provided (from args) order but installs dependencies first Implementation detail No order guarantee if you need this order, use setup requires Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
  • 97. 2 - How does pip perform an installation ? Installation order tries to respect the provided (from args) order but installs dependencies first Implementation detail No order guarantee if you need this order, use setup requires setup requires currently ”owned” by setuptools Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
  • 98. 2 - How does pip perform an installation ? Installation order tries to respect the provided (from args) order but installs dependencies first Implementation detail No order guarantee if you need this order, use setup requires setup requires currently ”owned” by setuptools might call easy install Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
  • 99. 2 - How does pip perform an installation ? Installation order tries to respect the provided (from args) order but installs dependencies first Implementation detail No order guarantee if you need this order, use setup requires setup requires currently ”owned” by setuptools might call easy install hopefully, pip might take control of this feature Xavier Fernandez (Polyconseil) pip internals July 20, 2015 20 / 38
  • 100. Outline for section 3 1 How does pip resolve dependencies ? 2 How does pip perform an installation ? 3 How does pip select the packages ? Xavier Fernandez (Polyconseil) pip internals July 20, 2015 21 / 38
  • 101. 3 - How does pip select the packages ? Context pip install foo bar Xavier Fernandez (Polyconseil) pip internals July 20, 2015 22 / 38
  • 102. 3 - How does pip select the packages ? Context pip install foo bar https://some pypi.com/simple/foo bar/foo bar-1.2.tar.gz https://some pypi.com/simple/foo bar/foo bar-1.0-py3-none-any.whl /home/user/wheelhouse/foo bar-3.0-py3-none-any.whl dozens of others... Xavier Fernandez (Polyconseil) pip internals July 20, 2015 22 / 38
  • 103. 3 - How does pip select the packages ? Context pip install foo bar https://some pypi.com/simple/foo bar/foo bar-1.2.tar.gz https://some pypi.com/simple/foo bar/foo bar-1.0-py3-none-any.whl /home/user/wheelhouse/foo bar-3.0-py3-none-any.whl dozens of others... What file should pip use ? different versions Xavier Fernandez (Polyconseil) pip internals July 20, 2015 22 / 38
  • 104. 3 - How does pip select the packages ? Context pip install foo bar https://some pypi.com/simple/foo bar/foo bar-1.2.tar.gz https://some pypi.com/simple/foo bar/foo bar-1.0-py3-none-any.whl /home/user/wheelhouse/foo bar-3.0-py3-none-any.whl dozens of others... What file should pip use ? different versions different formats Xavier Fernandez (Polyconseil) pip internals July 20, 2015 22 / 38
  • 105. 3 - How does pip select the packages ? Context pip install foo bar https://some pypi.com/simple/foo bar/foo bar-1.2.tar.gz https://some pypi.com/simple/foo bar/foo bar-1.0-py3-none-any.whl /home/user/wheelhouse/foo bar-3.0-py3-none-any.whl dozens of others... What file should pip use ? different versions different formats different locations Xavier Fernandez (Polyconseil) pip internals July 20, 2015 22 / 38
  • 106. 3 - How does pip select the packages ? Where the magic happens: pip/index.py PackageFinder.find requirement(req, upgrade) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 23 / 38
  • 107. 3 - How does pip select the packages ? Where the magic happens: pip/index.py PackageFinder.find requirement(req, upgrade) req is an InstallRequirement: django, django==1.8.3, Django<1.8 Xavier Fernandez (Polyconseil) pip internals July 20, 2015 23 / 38
  • 108. 3 - How does pip select the packages ? Where the magic happens: pip/index.py PackageFinder.find requirement(req, upgrade) req is an InstallRequirement: django, django==1.8.3, Django<1.8 upgrade is a boolean (--upgrade option) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 23 / 38
  • 109. 3 - How does pip select the packages ? Where the magic happens: pip/index.py PackageFinder.find requirement(req, upgrade) req is an InstallRequirement: django, django==1.8.3, Django<1.8 upgrade is a boolean (--upgrade option) returns an InstallationCandidate (a file/location) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 23 / 38
  • 110. 3 - How does pip select the packages ? find requirement(req, upgrade) calls self. find all versions(req.name) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
  • 111. 3 - How does pip select the packages ? find requirement(req, upgrade) calls self. find all versions(req.name) for django on PyPI, 112 InstallationCandidate Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
  • 112. 3 - How does pip select the packages ? find requirement(req, upgrade) calls self. find all versions(req.name) for django on PyPI, 112 InstallationCandidate on my setup (with private mirror and wheelhouse), 249 Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
  • 113. 3 - How does pip select the packages ? find requirement(req, upgrade) calls self. find all versions(req.name) for django on PyPI, 112 InstallationCandidate on my setup (with private mirror and wheelhouse), 249 filter on version requirements (==1.8.3) and --pre option Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
  • 114. 3 - How does pip select the packages ? find requirement(req, upgrade) calls self. find all versions(req.name) for django on PyPI, 112 InstallationCandidate on my setup (with private mirror and wheelhouse), 249 filter on version requirements (==1.8.3) and --pre option (stable) sort by version Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
  • 115. 3 - How does pip select the packages ? find requirement(req, upgrade) calls self. find all versions(req.name) for django on PyPI, 112 InstallationCandidate on my setup (with private mirror and wheelhouse), 249 filter on version requirements (==1.8.3) and --pre option (stable) sort by version returns most recent/currently installed version (--upgrade option) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 24 / 38
  • 116. 3 - How does pip select the packages ? find all versions: Three main sources indexes: --index-url, --extra-index-url or --no-index options Xavier Fernandez (Polyconseil) pip internals July 20, 2015 25 / 38
  • 117. 3 - How does pip select the packages ? find all versions: Three main sources indexes: --index-url, --extra-index-url or --no-index options find links: --find-links options Xavier Fernandez (Polyconseil) pip internals July 20, 2015 25 / 38
  • 118. 3 - How does pip select the packages ? find all versions: Three main sources indexes: --index-url, --extra-index-url or --no-index options find links: --find-links options dependency links: provided by some setup.py if --process-dependency-links Xavier Fernandez (Polyconseil) pip internals July 20, 2015 25 / 38
  • 119. 3 - How does pip select the packages ? Indexes expects something like scheme://some index/ Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
  • 120. 3 - How does pip select the packages ? Indexes expects something like scheme://some index/ goes to scheme://some index/package name to list available files Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
  • 121. 3 - How does pip select the packages ? Indexes expects something like scheme://some index/ goes to scheme://some index/package name to list available files Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
  • 122. 3 - How does pip select the packages ? Indexes expects something like scheme://some index/ goes to scheme://some index/package name to list available files Normalized name python-dateutil Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
  • 123. 3 - How does pip select the packages ? Indexes expects something like scheme://some index/ goes to scheme://some index/package name to list available files Normalized name python-dateutil Python-Dateutil Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
  • 124. 3 - How does pip select the packages ? Indexes expects something like scheme://some index/ goes to scheme://some index/package name to list available files Normalized name python-dateutil Python-Dateutil python dateutil Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
  • 125. 3 - How does pip select the packages ? Indexes expects something like scheme://some index/ goes to scheme://some index/package name to list available files Normalized name python-dateutil Python-Dateutil python dateutil pYthOn datEutiL Xavier Fernandez (Polyconseil) pip internals July 20, 2015 26 / 38
  • 126. 3 - How does pip select the packages ? With modern indexes - PyPI GET on https://pypi/simple/Python Dateutil Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
  • 127. 3 - How does pip select the packages ? With modern indexes - PyPI GET on https://pypi/simple/Python Dateutil redirects to https://pypi/simple/python-dateutil Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
  • 128. 3 - How does pip select the packages ? With modern indexes - PyPI GET on https://pypi/simple/Python Dateutil redirects to https://pypi/simple/python-dateutil re-redirects to https://pypi/simple/python-dateutil/ Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
  • 129. 3 - How does pip select the packages ? With modern indexes - PyPI GET on https://pypi/simple/Python Dateutil redirects to https://pypi/simple/python-dateutil re-redirects to https://pypi/simple/python-dateutil/ Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
  • 130. 3 - How does pip select the packages ? With modern indexes - PyPI GET on https://pypi/simple/Python Dateutil redirects to https://pypi/simple/python-dateutil re-redirects to https://pypi/simple/python-dateutil/ Older/simpler indexes GET on https://pypi/simple/Python Dateutil returns a 404 Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
  • 131. 3 - How does pip select the packages ? With modern indexes - PyPI GET on https://pypi/simple/Python Dateutil redirects to https://pypi/simple/python-dateutil re-redirects to https://pypi/simple/python-dateutil/ Older/simpler indexes GET on https://pypi/simple/Python Dateutil returns a 404 try GET on https://pypi/simple/ - (2,7M for PyPI) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
  • 132. 3 - How does pip select the packages ? With modern indexes - PyPI GET on https://pypi/simple/Python Dateutil redirects to https://pypi/simple/python-dateutil re-redirects to https://pypi/simple/python-dateutil/ Older/simpler indexes GET on https://pypi/simple/Python Dateutil returns a 404 try GET on https://pypi/simple/ - (2,7M for PyPI) match all the links of the index on the normalized form Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
  • 133. 3 - How does pip select the packages ? With modern indexes - PyPI GET on https://pypi/simple/Python Dateutil redirects to https://pypi/simple/python-dateutil re-redirects to https://pypi/simple/python-dateutil/ Older/simpler indexes GET on https://pypi/simple/Python Dateutil returns a 404 try GET on https://pypi/simple/ - (2,7M for PyPI) match all the links of the index on the normalized form Note: this is only performed on the first index but used on all Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
  • 134. 3 - How does pip select the packages ? With modern indexes - PyPI GET on https://pypi/simple/Python Dateutil redirects to https://pypi/simple/python-dateutil re-redirects to https://pypi/simple/python-dateutil/ Older/simpler indexes GET on https://pypi/simple/Python Dateutil returns a 404 try GET on https://pypi/simple/ - (2,7M for PyPI) match all the links of the index on the normalized form Note: this is only performed on the first index but used on all Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
  • 135. 3 - How does pip select the packages ? With modern indexes - PyPI GET on https://pypi/simple/Python Dateutil redirects to https://pypi/simple/python-dateutil re-redirects to https://pypi/simple/python-dateutil/ Older/simpler indexes GET on https://pypi/simple/Python Dateutil returns a 404 try GET on https://pypi/simple/ - (2,7M for PyPI) match all the links of the index on the normalized form Note: this is only performed on the first index but used on all New locations https://pypi/simple/ Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
  • 136. 3 - How does pip select the packages ? With modern indexes - PyPI GET on https://pypi/simple/Python Dateutil redirects to https://pypi/simple/python-dateutil re-redirects to https://pypi/simple/python-dateutil/ Older/simpler indexes GET on https://pypi/simple/Python Dateutil returns a 404 try GET on https://pypi/simple/ - (2,7M for PyPI) match all the links of the index on the normalized form Note: this is only performed on the first index but used on all New locations https://pypi/simple/ https://pypi/simple/python-dateutil Xavier Fernandez (Polyconseil) pip internals July 20, 2015 27 / 38
  • 137. 3 - How does pip select the packages ? indexes index links find links dependency links find url name Xavier Fernandez (Polyconseil) pip internals July 20, 2015 28 / 38
  • 138. 3 - How does pip select the packages ? indexes index links find links dependency links find url name sort locations file locations url locations Xavier Fernandez (Polyconseil) pip internals July 20, 2015 28 / 38
  • 139. 3 - How does pip select the packages ? indexes index links find links dependency links find url name sort locations file locations url locations Xavier Fernandez (Polyconseil) pip internals July 20, 2015 28 / 38
  • 140. 3 - How does pip select the packages ? indexes index links find links dependency links find url name sort locations file locations url locations Xavier Fernandez (Polyconseil) pip internals July 20, 2015 28 / 38
  • 141. 3 - How does pip select the packages ? sort locations if the scheme == file or if the path exists: if is a file: if html: -> urls else: -> files elif is a directory: if dealing with find_links: for all files: if html: -> urls else: -> files else: -> urls else: -> urls Xavier Fernandez (Polyconseil) pip internals July 20, 2015 29 / 38
  • 142. 3 - How does pip select the packages ? indexes index links find links dependency links find url name sort locations file locations url locations Xavier Fernandez (Polyconseil) pip internals July 20, 2015 30 / 38
  • 143. 3 - How does pip select the packages ? indexes index links find links dependency links find url name sort locations file locations url locations find links locations Xavier Fernandez (Polyconseil) pip internals July 20, 2015 30 / 38
  • 144. 3 - How does pip select the packages ? indexes index links find links dependency links find url name sort locations file locations url locations find links locations dep links locations Xavier Fernandez (Polyconseil) pip internals July 20, 2015 30 / 38
  • 145. 3 - How does pip select the packages ? indexes index links find links dependency links find url name sort locations file locations url locations find links locations dep links locations validate secure origin Xavier Fernandez (Polyconseil) pip internals July 20, 2015 30 / 38
  • 146. 3 - How does pip select the packages ? validate secure origin for url locations protocol, hostname, port (”https”, ”*”, ”*”), (”*”, ”localhost”, ”*”), (”*”, ”127.0.0.0/8”, ”*”), (”*”, ”::1/128”, ”*”), (”file”, ”*”, None), Xavier Fernandez (Polyconseil) pip internals July 20, 2015 31 / 38
  • 147. 3 - How does pip select the packages ? validate secure origin for url locations protocol, hostname, port (”https”, ”*”, ”*”), (”*”, ”localhost”, ”*”), (”*”, ”127.0.0.0/8”, ”*”), (”*”, ”::1/128”, ”*”), (”file”, ”*”, None), (”*”, host, ”*”), for all hosts from --trusted-host option Xavier Fernandez (Polyconseil) pip internals July 20, 2015 31 / 38
  • 148. 3 - How does pip select the packages ? indexes index links find links dependency links find url name sort locations file locations url locations find links locations dep links locations validate secure origin Xavier Fernandez (Polyconseil) pip internals July 20, 2015 32 / 38
  • 149. 3 - How does pip select the packages ? indexes index links find links dependency links find url name sort locations file locations url locations find links locations dep links locations validate secure origin HTMLPage Xavier Fernandez (Polyconseil) pip internals July 20, 2015 32 / 38
  • 150. 3 - How does pip select the packages ? HTMLPage for urls looking like archives, check content type first if ”text/html”, continue get content and parse check all links if rel contains ’homepage’ or ’download’ add it (untrusted to url locations) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 33 / 38
  • 151. 3 - How does pip select the packages ? HTMLPage for urls looking like archives, check content type first if ”text/html”, continue get content and parse check all links if rel contains ’homepage’ or ’download’ add it (untrusted to url locations) Note with api version 2 i.e. PyPI, <a> can be marked with rel="internal" only trust internals <meta name="api-version" value="2" /> in the page head --allow-external/--allow-all-external Xavier Fernandez (Polyconseil) pip internals July 20, 2015 33 / 38
  • 152. 3 - How does pip select the packages ? indexes index links find links dependency links find url name sort locations file locations url locations find links locations dep links locations validate secure origin HTMLPage Xavier Fernandez (Polyconseil) pip internals July 20, 2015 34 / 38
  • 153. 3 - How does pip select the packages ? indexes index links find links dependency links find url name sort locations file locations url locations find links locations dep links locations validate secure origin HTMLPage package versions links Xavier Fernandez (Polyconseil) pip internals July 20, 2015 34 / 38
  • 154. 3 - How does pip select the packages ? package versions filter push egg links (containing #egg=) to the end Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
  • 155. 3 - How does pip select the packages ? package versions filter push egg links (containing #egg=) to the end select/reject on lots of criteria Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
  • 156. 3 - How does pip select the packages ? package versions filter push egg links (containing #egg=) to the end select/reject on lots of criteria Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
  • 157. 3 - How does pip select the packages ? package versions filter push egg links (containing #egg=) to the end select/reject on lots of criteria Criteria unsupported format (like bz2 on some install) Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
  • 158. 3 - How does pip select the packages ? package versions filter push egg links (containing #egg=) to the end select/reject on lots of criteria Criteria unsupported format (like bz2 on some install) macos10x files ending in .zip Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
  • 159. 3 - How does pip select the packages ? package versions filter push egg links (containing #egg=) to the end select/reject on lots of criteria Criteria unsupported format (like bz2 on some install) macos10x files ending in .zip --only-binary/--no-binary options Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
  • 160. 3 - How does pip select the packages ? package versions filter push egg links (containing #egg=) to the end select/reject on lots of criteria Criteria unsupported format (like bz2 on some install) macos10x files ending in .zip --only-binary/--no-binary options for wheel files: check compatibility and for non-windows/macos installs, refuse platform specific wheels from PyPI Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
  • 161. 3 - How does pip select the packages ? package versions filter push egg links (containing #egg=) to the end select/reject on lots of criteria Criteria unsupported format (like bz2 on some install) macos10x files ending in .zip --only-binary/--no-binary options for wheel files: check compatibility and for non-windows/macos installs, refuse platform specific wheels from PyPI tries to extract version from filename and reject file if fails Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
  • 162. 3 - How does pip select the packages ? package versions filter push egg links (containing #egg=) to the end select/reject on lots of criteria Criteria unsupported format (like bz2 on some install) macos10x files ending in .zip --only-binary/--no-binary options for wheel files: check compatibility and for non-windows/macos installs, refuse platform specific wheels from PyPI tries to extract version from filename and reject file if fails if version contains -py2/-py3/-py27/etc, check python version Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
  • 163. 3 - How does pip select the packages ? package versions filter push egg links (containing #egg=) to the end select/reject on lots of criteria Criteria unsupported format (like bz2 on some install) macos10x files ending in .zip --only-binary/--no-binary options for wheel files: check compatibility and for non-windows/macos installs, refuse platform specific wheels from PyPI tries to extract version from filename and reject file if fails if version contains -py2/-py3/-py27/etc, check python version check internal/external hosted files and verifiable/unverifiable files --allow-external/--allow-all-external/--allow-unverified Xavier Fernandez (Polyconseil) pip internals July 20, 2015 35 / 38
  • 164. 3 - How does pip select the packages ? Preference Order file sources are prefered Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
  • 165. 3 - How does pip select the packages ? Preference Order file sources are prefered then direct find links files Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
  • 166. 3 - How does pip select the packages ? Preference Order file sources are prefered then direct find links files then urls links Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
  • 167. 3 - How does pip select the packages ? Preference Order file sources are prefered then direct find links files then urls links direct dependency links files Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
  • 168. 3 - How does pip select the packages ? Preference Order file sources are prefered then direct find links files then urls links direct dependency links files this is what find all versions returns Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
  • 169. 3 - How does pip select the packages ? Preference Order file sources are prefered then direct find links files then urls links direct dependency links files this is what find all versions returns Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
  • 170. 3 - How does pip select the packages ? Preference Order file sources are prefered then direct find links files then urls links direct dependency links files this is what find all versions returns Note Wheel cache (pip 7 feature) has no impact on find all versions Xavier Fernandez (Polyconseil) pip internals July 20, 2015 36 / 38
  • 171. 3 - How does pip select the packages ? find all versions(’pep8’) [<InstallationCandidate(’pep8’, <Version(’1.5.7’)>, <Link file:///home/user/wheelhouse/pep8-1.5.7-py2.py3-none-any.whl>)>, <InstallationCandidate(’pep8’, <Version(’1.5.0’)>, <Link file:///home/user/wheelhouse/pep8-1.5.0-py27-none-any.whl>)>, ... <InstallationCandidate(’pep8’, <Version(’0.3.1’)>, <Link https://private_pypi/simple/pep8/pep8-0.3.1.tar.gz>)>, <InstallationCandidate(’pep8’, <Version(’0.4.1’)>, <Link https://private_pypi/simple/pep8/pep8-0.4.1.tar.gz>)>, ... <InstallationCandidate(’pep8’, <Version(’1.5.2’)>, <Link https://pypi.python.org/packages/3.3/p/pep8/ pep8-1.5.2-py2.py3-none-any.whl#md5=a0caba277a2f491b1634246a338a1235>)>, <InstallationCandidate(’pep8’, <Version(’1.5.3’)>, <Link https://pypi.python.org/packages/3.3/p/pep8/ pep8-1.5.3-py2.py3-none-any.whl#md5=0654904760aa9a24062bf367f39e873e>)>] Xavier Fernandez (Polyconseil) pip internals July 20, 2015 37 / 38