This talk showcases various example scenarios around the ins/outs of managing Python packages. Scenarios range from intra-package problems of how to build a package, inter-package problems of how to connect packages, to general usage/environment/setup problems that developers have a hard time debugging. For each of these scenarios, I'll highlight the pros/cons, best practices, and how to overcome the issues developers commonly face. Some examples: - installing packages from public, local file system, git (ssh + https), and how this affects deployment procedures - handling systems with multiple Python versions or package managers - how app structures and imports can affect testability - relative vs absolute imports - virtual environments - setup scripts - package versioning (why, when, how) - how new code changes could not be taking effect (old .pyc files, or updated import not catching) - handling sub-packages/dependencies that utilize parent configuration files - working with PYTHONPATH - executing python as modules or scripts Talk presented on Nov 18, 2017 at PyCon Canada 2017.

1. Effective Python Package Management
Devon Bernard
VP of Engineering @ Enlitic
@devonwbernard

2. Intra-Package Inter-Package
Types of Package Management

3. Intra-Package

4. Configuration Files
Why
1. Centralizing variables into one file makes them easier to find
2. Environment related variables don’t need to be juggled in git
3. Safely use secret variables (e.g. passwords) without leaking into git
database = 'production_url'
File.py (production)
~~ database = ’localhost_url'
++ def ban_user(email):
++ ...
File.py (local machine)

5. Don’t Leak Passwords

6. Configuration File Setup
import yaml
yaml_config = yaml.load(open(('../my-app.yaml'))
print yaml_config[’database’]
Usage
# DEFAULT VALUES
database: postgresql://production:5432/my_db
# LOCAL OVERRIDES
database: postgresql://localhost:5432/my_db
my-app.yaml.example
my-app.yaml

7. App Configurations
class Config(object):
DEBUG = False
CSRF_ENABLED = True
SECRET = yaml_config['flask_login_secret']
SQLALCHEMY_DATABASE_URI = yaml_config['database']
class DevelopmentConfig(Config):
DEBUG = True
class TestingConfig(Config):
TESTING = True
SQLALCHEMY_DATABASE_URI = yaml_config['test_database’]
DEBUG = True
class ProductionConfig(Config):
DEBUG = False
TESTING = False
app_config = {
'development': DevelopmentConfig,
'testing': TestingConfig,
'production': ProductionConfig
}
app = create_app('development')
app.run()
run.py
conftest.py
@pytest.fixture(scope='session')
def mock_app():
mock = create_app(’testing')
cxt = mock.app_context()
cxt.push()
yield mock

8. App Factories
app/__init__.py
db = SQLAlchemy()
def create_app(config_name):
app = Flask(__name__, instance_relative_config=True)
app.config.from_object(app_config[config_name])
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.secret_key = app.config['SECRET']
cors = CORS(app, supports_credentials=True)
db.init_app(app)
login_manager.init_app(app)
app.register_blueprint(api)
return app

9. App Factories (2)
app/__init__.py
from flask import Flask
app = Flask(__name__)
import api
import db
app/api/__init__.py
from app import app, db
import endpoint_group1
import endpoint_group2
app/db/__init__.py
from app.utils.config import config
from sqlalchemy.orm import scoped_session
conn = scoped_session(config...)
import query_group1
import query_group2
Issues:
• Circular imports (prevents testing)
• Lack of separation of concerns
between app module and app object

10. Package Versioning
Why
• Stability - New updates won’t break users’ existing builds
• Planned Obsolescence - Warn users of features getting deprecated
• Transparency - Enables better documentation of change logs
How
• Setup.py / setuptools / twine
• BumpVersion / Versioneer / ( Zest.releaser )

11. Versioning Scheme
PEP440 standard from packaging.python.org
1.2.0.dev1 # Development release
1.2.0a1 # Alpha Release
1.2.0b1 # Beta Release
1.2.0rc1 # Release Candidate
1.2.0 # Final Release
1.2.0.post1 # Post Release
15.10 # Date based release
23 # Serial release
[MAJOR].[MINOR](.[MAINTENANCE])(.[CYCLE][N])
Major - Not backwards compatible or many changes
Minor - Added functionality
Maintenance/Micro - Bug fixes (or features if long version cycle)

12. Setup.py
from setuptools import setup, find_packages
setup(
name='My Project',
version='0.1.0',
description='',
author='PERSON/COMPANY',
author_email='you@example.com',
url='https://github.com/USERNAME/REPO',
packages=find_packages(),
install_requires=[
'alembic==0.8.6',
'psycopg2==2.6.1',
'PyYAML==3.11',
'SQLAlchemy==1.0.14'
]
)
setup.py (PEP440)
python setup.py install
Install package on your local environment
python setup.py register
Reserve package name on the Python Package Index (PyPI)
python setup.py sdist
Create a source-distribution bundle
twine upload dist/*
Push your currently version to PyPI

13. Test PyPI
Want to try distributions in a risk-free environment?
https://testpypi.python.org/

14. Inter-Package

15. Where Are My Packages?
$ python run.py
ImportError: No module named flask_sqlalchemy
$ pip install flask_sqlalchemy
$ python run.py
ImportError: No module named smtplib
$ pip install smtplib
$ python run.py
ImportError: No module named passlib
$ ... #FacePalm

16. Requirements.txt
$ python run.py
ImportError: No module named flask_sqlalchemy
$ pip install –r requirements.txt
Collecting flask_sqlalchemy
Collecting smtplib
Collecting passlib
...
$ python run.py
* Running on http://127.0.0.1:5000/
Flask == 0.12
Flask-Cors == 3.0.3
Flask-SQLAlchemy == 2.1
Flask-Login == 0.3.2
psycopg2 == 2.6.1
PyYAML == 3.11
requests == 2.10.0
SQLAlchemy == 1.0.14
passlib == 1.6.5
bcrypt == 3.1.1
WTForms == 2.1
...
requirements.txt

17. Requirements.txt & Private Git Packages
-e git+https://github.com/MY_USER/MY_REPO.git#egg=MY_REPO # Will ask for password
-e git+ssh://git@github.com/MY_USER/MY_REPO.git#egg=MY_REPO
-e git+git://github.com/MY_USER/MY_REPO.git#egg=MY_REPO
Pip supports cloning over git, git+https, and git+ssh
-e git+ssh://git@github.com/MY_USER/MY_REPO.git@staging#egg=MY_REPO
-e git+ssh://git@github.com/MY_USER/MY_REPO.git@da39a3ee#egg=MY_REPO
Install packages from commit hash or non-master branch
If you want to add git references in setup.py, use dependency_links

18. Requirements.txt & Local Packages
git+file:///PATH/TO/MY/PROJECT
Install packages from local file system
Why use this method?
• Ability to quickly test whether changes to another python project worked without having to push to github
• This flow is fully local and keeps your git history cleaner
WARNING: When using the local file system method, your changes must be locally committed otherwise pip will
not notice the changes.

19. Global PIP Cache
Using the Wrong Dependencies?
Project X
(Flask==0.12)
Project Y
(Flask==0.8)
Common Package Management
Problems
• Need to use different versions of the
same package on another project or
branch
• Project broke because it was using a
package version overrode by
another project
• Requirements.txt is missing an entry
because previous developer already
had the package installed
Project
Z
Project
W

20. ~/.venvs/Z
~/.venvs/Y
~/.venvs/W
~/.venvs/X
Virtual Environments
Project X
(Flask==0.12)
Project Y
(Flask==0.8)
Project
Z
Project
W

21. Virtual Environments (2)
Setup
Issues?
Just delete your venv and create a fresh one
$ virtualenv ~/.venvs/X
$ source ~/.venvs/X/bin/activate
(X)$ pip install -r requirements.txt
$ rm –rf ~/.venvs/X
Clean & easy dependency
management; especially when
working on multiple projects
Why?
• Package version conflicts
• v5.0.1 != v5.3.0
• Dependency contamination
across multiple projects
• Testing dependency upgrades

22. Common Scenario
1. Some files are changed
2. The code is re-run
3. The changes don’t appear to exist or are being ignored
Potential Causes
• Using an old version of a dependency
Solution: Check which version of dependencies are currently installed (`pip freeze`)
• Using old .pyc files
Solution: Remove all .pyc files in your project
Python Ignoring New Code Changes?

23. Common .pyc conflict scenarios:
• Moved or renamed python directories
• Moved or renamed python files
• Added or removed __init__.py files
• Created a variable with the same name as a module/file
To clear out all .pyc files and start fresh, run the following:
Removing old .pyc files
$ find . -name '*.pyc' -delete

24. Devon Bernard
VP of Engineering @ Enlitic
dwbcoding@gmail.com
@devonwbernard
Thank you!
Any questions?