This document discusses managing Python dependencies at Beeswax using Bazel and related tools. It has 30 engineers working on a monorepo of 100 applications totaling 300 kLoC of Python code. Previously they used a global requirements.txt file, pipenv, and rules_python, but now use Bazel, pip-tools, and virtualenvs to manage dependencies. This new approach provides fully correct and minimal dependencies across the monorepo, works well with the monorepo structure, and maintains a history of dependency changes through lock files, though it has made their development and CI workflows more complex.

1. Python Dependencies
The Right* Way
Bazel + Python @ Beeswax
Ron Rothman
* For us. YMMV.

2. ● 30 engineers
● monorepo
● 100 applications
● 300 kLoPyC
1.Global requirements.txt
2.rules_python
3.pipenv
We have: We tried:
● bazel test
● bazel run
● deployments
We need deps for:

3. Bazel
pip-tools
Virtualenvs
Wheels
Beeve
lockfiles

4. # requirements.lock
boto3==1.10.2
botocore==1.13.2 # via boto3, s3transfer
certifi==2019.9.11 # via requests
chardet==3.0.4 # via requests
docutils==0.15.2 # via botocore
futures==3.3.0 # via s3transfer
idna==2.8 # via requests
jmespath==0.9.4 # via boto3, botocore
python-dateutil==2.8.0 # via botocore
requests==2.22.0
s3transfer==0.2.1 # via boto3
six==1.12.0 # via python-dateutil
urllib3==1.25.6 # via botocore,
requests
py_binary(
name = "my_app",
deps = ["lib_a"],
requires = ["requests>=2.21"],
)
py_library(
name = "lib_a",
deps = ["lib_b"],
requires = ["requests>=2.18"],
)
py_library(
name = "lib_b",
deps = ["lib_c"],
requires = ["boto3~=1.9"],
)
py_venv(
name = "my_app_env",
targets = [":my_app"],
)
# requirements.in
boto3~=1.9
requests>=2.18
requests>=2.21

5. Needs improvement:
● Dev/CI workflow more complicated than before
● Bazel cache misses
● Dependency changes are not propagated automatically
Working well:
● 100% correct (and minimal) dependencies!
● Monorepo-friendly
● History of dependency changes (lock files)
● Wheels are easy

6. Thank You 🙏🏻
ron {at} beeswax.com