1. Developing in Python
on Red Hat platforms
Nick Coghlan
Senior Software Engineer
Graham Dumpleton
Developer Advocate for OpenShift
June 28th
2016

2. Using Python on Red Hat Platforms
● Python for Network Services
● Python for Applications
● Python for System Administration

3. Current Transitions in the Python Ecosystem
● Migration to Python 3
● Modernizing the Python 2.7 Network Security Stack
● Defaulting to HTTPS Certificate Verification

4. Python for Network Services

12. $ oc new-app https://gitlab.com/osevg/python-django-modwsgi.git
--> Found image 772dc19 (4 weeks old) in image stream "python" in project "openshift" under tag "3.4" for "python"
Python 3.4
----------
Platform for building and running Python 3.4 applications
Tags: builder, python, python34, rh-python34
* The source repository appears to match: python
* A source build using source code from https://gitlab.com/osevg/python-django-modwsgi.git will be created
* The resulting image will be pushed to image stream "python-django-modwsgi:latest"
* This image will be deployed in deployment config "python-django-modwsgi"
* Port 8080/tcp will be load balanced by service "python-django-modwsgi"
* Other containers can access this service through the hostname "python-django-modwsgi"
--> Creating resources with label app=python-django-modwsgi ...
imagestream "python-django-modwsgi" created
buildconfig "python-django-modwsgi" created
deploymentconfig "python-django-modwsgi" created
service "python-django-modwsgi" created
--> Success
Build scheduled, use 'oc logs -f bc/python-django-modwsgi' to track its progress.
Run 'oc status' to view your app.
$ oc expose svc/python-django-modwsgi
route "python-django-modwsgi" exposed

13. Source to Image (S2I)
https://github.com/openshift/source-to-image

14. $ s2i build https://gitlab.com/osevg/python-django-modwsgi.git centos/python-34-centos7 python-django-modwsgi
I0610 10:02:07.422470 84805 docker.go:352] Image "centos/python-34-centos7:latest" not available locally, pulling ...
I0610 10:02:36.501637 84805 clone.go:32] Downloading "https://gitlab.com/osevg/python-django-modwsgi.git" ...
I0610 10:02:38.831859 84805 install.go:251] Using "assemble" installed from "image:///usr/libexec/s2i/assemble"
I0610 10:02:38.831913 84805 install.go:251] Using "run" installed from "image:///usr/libexec/s2i/run"
I0610 10:02:38.831943 84805 install.go:251] Using "save-artifacts" installed from "image:///usr/libexec/s2i/save-artifacts"
I0610 10:02:38.832495 84805 environment.go:60] Setting 1 environment variables provided by environment file in sources
---> Copying application source ...
---> Installing dependencies ...
...
---> Collecting Django static files ...
I0610 10:03:00.876021 84805 environment.go:60] Setting 1 environment variables provided by environment file in sources
$ docker run --rm -p 8080:8080 python-django-modwsgi
---> Running application from Python script (app.py) ...
[Fri Jun 10 00:07:33.580013 2016] [mpm_event:notice] [pid 1:tid 139758878566464] AH00489: Apache/2.4.6 (CentOS)
mod_wsgi/4.5.2 Python/3.4.2 configured -- resuming normal operations
[Fri Jun 10 00:07:33.580149 2016] [core:notice] [pid 1:tid 139758878566464] AH00094: Command line: 'httpd (mod_wsgi-
express) -f /tmp/mod_wsgi-localhost:8080:1001/httpd.conf -D MOD_WSGI_MULTIPROCESS -D
MOD_WSGI_WITH_PROXY_HEADERS -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D
MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D
MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'

15. Docker Base Images (S2I Enabled)
● RHEL 7
– Python 2.7 → registry.access.redhat.com/rhscl/python-27-rhel7
– Python 3.3 → registry.access.redhat.com/openshift3/python-33-rhel7
– Python 3.4 → registry.access.redhat.com/rhscl/python-34-rhel7
● CentOS 7
– Python 2.7 → docker.io/centos/python-27-centos7
– Python 3.3 → docker.io/openshift/python-33-centos7
– Python 3.4 → docker.io/centos/python-34-centos7

16. Migrating from OpenShift 2 to 3
● Converting of existing applications.
● Backward compatible S2I builder.
● Guidelines for porting applications.
● Templates to aid in porting applications.

18. Python for Applications

19. Why *Not* Containers?
● Containers are the recommended option for network services
● However:
– Container support for rich desktop applications is currently limited
– Container runtime may impose unwanted overhead on dedicated systems
– Containers may give more isolation than is wanted
– Applications may require non-trivial modification to run as a privileged container
● Software Collections aim to offer “minimum viable runtime isolation”
– Add new executable directories to front of PATH
– Add new shared library directories to front of LD_LIBRARY_PATH

20. Why Software Collections for Python?
● Use newer Python runtimes without impacting system components
● Use a common Python runtime across multiple operating system versions
● Python 3 for Red Hat Enterprise Linux 6 & 7!

21. Red Hat Software Collections
● Platforms
– Red Hat Enterprise Linux 6 & Red Hat Enterprise Linux 7
– CentOS 6 and 7 (via softwarecollections.org)
– Basis for OpenShift language runtimes
● Available versions (as of RHSCL 2.2)
– Python 2.7.8 (+ selected backports)
– Python 3.5.1 (+ selected backports)
– Python 3.4.2 (+ selected backports)
– Python 3.3.2 (+ selected backports)

22. Python Virtual Environments
● Software Collections allow multiple runtimes to share a system without conflicting
● Virtual environments allow multiple Python applications to share a runtime
● Low or no runtime overhead: just add/replace directories in Python’s import path
● Cleanly isolate application dependencies from platform components
● Dependencies within the environment managed with pip
● Created via:
– python3 -m venv (Python 3.4+)
– virtualenv (Python 2.7, Python 3.x)
● Not included in the Red Hat Enterprise Linux System Python

24. Constructing a Layered Application
● Base platform (via system package manager):
– Operating system (e.g. kernel, C runtime)
– Language runtime (from Software Collections)
– Other external dependencies (e.g. OpenSSL)
● Modify shared library loading (via enabled Software Collection)
● Modify Python import configuration (via virtualenv or standard library’s venv)
● Inside the virtual environment:
– Application dependencies (managed via pip)
– Application source code (managed via pip, or direct from source control)

25. Deploying a Layered Application (1)
● Application RPM with generated artifacts in SRPM:
– Create full installation in representative environment
– Bundle entire virtualenv and other desired components into SRPM
– Also include scripts to appropriately activate SCL and virtual environment
● Some generated files will end up in the SRPM
– Pre-compiled Python files
– Executable wrappers for pip managed Python scripts

26. Deploying a Layered Application (2)
● Application RPM with source-only SRPM:
– SRPM contains source for application and any application level dependencies
– virtual environment created and configured during RPM build process
● Not yet fully supported in pip
– some pip generated metadata will incorrectly include RPM buildroot paths
– shouldn’t matter for most RPM based deployment use cases

27. Managing Application Dependencies
● The Python Package Index is not an App Store!
● Designed to minimize barriers to publication:
– No pre-publication review
– Publisher Terms of Service ensure right to redistribute, not to run or modify
– Publishers may delete (but not replace) previously published versions
● Recommendation: use caching proxies and a component review process
– Commercially supported options: JFrog Artifactory, Sonatype Nexus
– Community/self-supported options: devpi, Python plugin for Pulp
– Check licensing, export restrictions, project governance, ...
● Security monitoring & response also becomes a dev team responsibility!

28. Python for System Administration

29. Why *Not* Software Collections?
● Containers are the recommended option for network services
● Software Collections are recommended for applications
● However:
– some platform bindings are only installed in the main System Python
– the System Python is available on all systems by default
● Some system administration tasks are best handled with the System Python

30. System Python
● Red Hat Enterprise Linux 6
– Python 2.6.6 (+ selected backports)
● Red Hat Enterprise Linux 7
– Python 2.7.5 (+ selected backports)
● Fedora 23 and later
– Recent Python 3.x (rebases rather than backports)
– Recent Python 2.7 available as system package, may not have all system bindings

31. Caveats and Challenges
● Restricted to features of System Python on oldest supported platform
● Community maintained libraries and frameworks often require newer runtimes
● Conflict between supporting:
– Red Hat Enterprise Linux 5 (Python 2.4 lacks Py3 forward compatibility features)
– Fedora 23+ (Python 3.x as System Python)
● May want to consider higher level system abstractions like Ansible

32. Migration to Python 3

33. Python 2.7 Support Timeline
● Anticipated community end-of-life for Python 2.7 in January 2020
– https://docs.python.org/devguide/#status-of-python-branches
● Supported in Red Hat Enterprise Linux 7 until June 2024
– https://access.redhat.com/support/policy/updates/errata
● Anticipate community project support for Python 2 declining sharply post-2020
– Already seeing new community projects starting as Python 3 only

34. Python 3 Migration Techniques
● General “refactoring enablement” techniques:
– automated regression testing frameworks
– static structural analysis tools
● Recommended approach for applications and network services:
– Migrate to the Python 2.7 SCL or OpenShift image (if using the system Python)
– Follow https://docs.python.org/3/howto/pyporting.html
– Migrate to the latest Python 3.x SCL or OpenShift image
● Recommended approach for system administration tools:
– Consider using Fedora 23+ to look for potential pain points

35. Python 3 Migration Notes
● Common subset of Python 2.6+ and 3.3+ is quite large
● Many deprecated idioms can be updated automatically
● Key data & workload driven pain points
– Explicit bytes/unicode separation
– Removal of implicit cross-type comparisons
● Automated refactoring and compatibility testing tools continue to improve

36. Modernizing the Python 2.7
Network Security Stack

37. New Security Features in Python 2.7
● https://docs.python.org/2/whatsnew/2.7.html#pep-466-network-security-enhancements-
for-python-2-7
● Constant-time comparison (hmac.compare_digest())
● Password storage hashing (hashlib.pbkdf2_hmac())
● ssl module rebase on Python 3.4 implementation
– Server Name Indication support
– SSLContext for SSL configuration
– Configuration support for TLS 1.x
– Access to system certificate stores
– ...

38. Availability in Red Hat Products
● Red Hat Enterprise Linux 7.2
– backported to System Python
● Red Hat Software Collections 2.0+
– default in Python 3.4
● Red Hat Software Collections 2.2+
– backported to Python 2.7
– default in Python 3.5

39. Third Party Module Compatibility
● ssl module rebase changed several private implementation details
● Some third party libraries had used internal APIs instead of requesting public ones
● Backport offers greater compatibility than upstream rebase
● Testing before upgrading is still recommended
● Report problems through the usual channels

40. Defaulting to HTTPS
Certificate Verification

41. What does “HTTPS” Mean?
● Historical Python standard library answer:
– “HTTP connection with SSL/TLS enabled”
– didn’t check certificate validity or remote host identification
● Modern Python standard library answer:
– “What web browsers say it means”
– still a HTTP connection with SSL/TLS enabled
– also checks for certificate validity
– also checks remote host identification against system certificate store

42. Verifying HTTPS Certificates
● https://docs.python.org/2/whatsnew/2.7.html#pep-476-enabling-certificate-verification-by-
default-for-stdlib-http-clients
● Default behavior of standard library HTTPS clients in:
– Python 2.7.9+
– Python 3.4.3+
– Python 3.5.0+
● Turns a silent security failure into a noisy connection failure
● Potential problems:
– Self-signed internal certificates
– Expired certificates
– Internal CAs not configured on client system

43. On Red Hat Platforms
● File-based opt-in:
– Config setting in /etc/python/cert-verification.cfg
– Red Hat Enterprise Linux 7.2+ System Python
– Red Hat Software Collections 2.2+ Python 2.7 collection
● Default behavior:
– Red Hat Software Collections Python 3.5 collection
● Details in Knowledge Base
– https://access.redhat.com/articles/2039753

44. Future Configuration Options
● https://docs.python.org/2/whatsnew/2.7.html#pep-493-https-verification-migration-tools
-for-python-2-7
● Adds new Python 2.7 specific configuration options
– ssl._https_verify_certificates() API
– PYTHONHTTPSVERIFY environment variable
● Can be used to revert Python 2.7.12+ to Python 2.7.8 behavior
● Python 2.7 only, not supported by any version of Python 3

45. Resources

46. OpenShift
● OpenShift
– https://www.openshift.com
● OpenShift Online Preview
– https://www.openshift.com/devpreview/register.html
● OpenShift Container Development Kit
– https://developers.redhat.com/products/cdk/overview/
● OpenShift Origin
– https://www.openshift.org
● OpenShift Origin All-In-One VM
– https://www.openshift.org/vm

47. Python
● Red Hat Software Collections
– http://developers.redhat.com/products/softwarecollections/
– Access: https://access.redhat.com/solutions/472793
● Software Collections Upstream
– https://wiki.centos.org/SpecialInterestGroup/SCLo
– https://www.softwarecollections.org

48. Related DevNation 2016 Sessions
● OpenShift
– OpenShift Enterprise 3 walk-through with Docker and Kubernetes
● Container Development Kit
– CDK 2.0: Docker, Kubernetes, and OSE on your desk
– Container development for command line developers
● Software Collections
– Software Collections: Easy access to the cutting edge