Developing in Python on Red Hat Platforms (DevNation 2016)

200 views

Published on

Overview by Graham Dumpleton (OpenShift) and Nick Coghlan (Red Hat Platform Engineering) of:

- available and recommended options for developing in Python on Red Hat Platforms for network service development, application development, and systems administration tasks
- current transitions in the Python ecosystem, including network security enhancements and the ongoing community migration to Python 3

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
200
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Developing in Python on Red Hat Platforms (DevNation 2016)

  1. 1. Developing in Python on Red Hat platforms Nick Coghlan Senior Software Engineer Graham Dumpleton Developer Advocate for OpenShift June 28th 2016
  2. 2. Using Python on Red Hat Platforms ● Python for Network Services ● Python for Applications ● Python for System Administration
  3. 3. Current Transitions in the Python Ecosystem ● Migration to Python 3 ● Modernizing the Python 2.7 Network Security Stack ● Defaulting to HTTPS Certificate Verification
  4. 4. Python for Network Services
  5. 5. $ oc new-app https://gitlab.com/osevg/python-django-modwsgi.git --> Found image 772dc19 (4 weeks old) in image stream "python" in project "openshift" under tag "3.4" for "python" Python 3.4 ---------- Platform for building and running Python 3.4 applications Tags: builder, python, python34, rh-python34 * The source repository appears to match: python * A source build using source code from https://gitlab.com/osevg/python-django-modwsgi.git will be created * The resulting image will be pushed to image stream "python-django-modwsgi:latest" * This image will be deployed in deployment config "python-django-modwsgi" * Port 8080/tcp will be load balanced by service "python-django-modwsgi" * Other containers can access this service through the hostname "python-django-modwsgi" --> Creating resources with label app=python-django-modwsgi ... imagestream "python-django-modwsgi" created buildconfig "python-django-modwsgi" created deploymentconfig "python-django-modwsgi" created service "python-django-modwsgi" created --> Success Build scheduled, use 'oc logs -f bc/python-django-modwsgi' to track its progress. Run 'oc status' to view your app. $ oc expose svc/python-django-modwsgi route "python-django-modwsgi" exposed
  6. 6. Source to Image (S2I) https://github.com/openshift/source-to-image
  7. 7. $ s2i build https://gitlab.com/osevg/python-django-modwsgi.git centos/python-34-centos7 python-django-modwsgi I0610 10:02:07.422470 84805 docker.go:352] Image "centos/python-34-centos7:latest" not available locally, pulling ... I0610 10:02:36.501637 84805 clone.go:32] Downloading "https://gitlab.com/osevg/python-django-modwsgi.git" ... I0610 10:02:38.831859 84805 install.go:251] Using "assemble" installed from "image:///usr/libexec/s2i/assemble" I0610 10:02:38.831913 84805 install.go:251] Using "run" installed from "image:///usr/libexec/s2i/run" I0610 10:02:38.831943 84805 install.go:251] Using "save-artifacts" installed from "image:///usr/libexec/s2i/save-artifacts" I0610 10:02:38.832495 84805 environment.go:60] Setting 1 environment variables provided by environment file in sources ---> Copying application source ... ---> Installing dependencies ... ... ---> Collecting Django static files ... I0610 10:03:00.876021 84805 environment.go:60] Setting 1 environment variables provided by environment file in sources $ docker run --rm -p 8080:8080 python-django-modwsgi ---> Running application from Python script (app.py) ... [Fri Jun 10 00:07:33.580013 2016] [mpm_event:notice] [pid 1:tid 139758878566464] AH00489: Apache/2.4.6 (CentOS) mod_wsgi/4.5.2 Python/3.4.2 configured -- resuming normal operations [Fri Jun 10 00:07:33.580149 2016] [core:notice] [pid 1:tid 139758878566464] AH00094: Command line: 'httpd (mod_wsgi- express) -f /tmp/mod_wsgi-localhost:8080:1001/httpd.conf -D MOD_WSGI_MULTIPROCESS -D MOD_WSGI_WITH_PROXY_HEADERS -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'
  8. 8. Docker Base Images (S2I Enabled) ● RHEL 7 – Python 2.7 → registry.access.redhat.com/rhscl/python-27-rhel7 – Python 3.3 → registry.access.redhat.com/openshift3/python-33-rhel7 – Python 3.4 → registry.access.redhat.com/rhscl/python-34-rhel7 ● CentOS 7 – Python 2.7 → docker.io/centos/python-27-centos7 – Python 3.3 → docker.io/openshift/python-33-centos7 – Python 3.4 → docker.io/centos/python-34-centos7
  9. 9. Migrating from OpenShift 2 to 3 ● Converting of existing applications. ● Backward compatible S2I builder. ● Guidelines for porting applications. ● Templates to aid in porting applications.
  10. 10. Python for Applications
  11. 11. Why *Not* Containers? ● Containers are the recommended option for network services ● However: – Container support for rich desktop applications is currently limited – Container runtime may impose unwanted overhead on dedicated systems – Containers may give more isolation than is wanted – Applications may require non-trivial modification to run as a privileged container ● Software Collections aim to offer “minimum viable runtime isolation” – Add new executable directories to front of PATH – Add new shared library directories to front of LD_LIBRARY_PATH
  12. 12. Why Software Collections for Python? ● Use newer Python runtimes without impacting system components ● Use a common Python runtime across multiple operating system versions ● Python 3 for Red Hat Enterprise Linux 6 & 7!
  13. 13. Red Hat Software Collections ● Platforms – Red Hat Enterprise Linux 6 & Red Hat Enterprise Linux 7 – CentOS 6 and 7 (via softwarecollections.org) – Basis for OpenShift language runtimes ● Available versions (as of RHSCL 2.2) – Python 2.7.8 (+ selected backports) – Python 3.5.1 (+ selected backports) – Python 3.4.2 (+ selected backports) – Python 3.3.2 (+ selected backports)
  14. 14. Python Virtual Environments ● Software Collections allow multiple runtimes to share a system without conflicting ● Virtual environments allow multiple Python applications to share a runtime ● Low or no runtime overhead: just add/replace directories in Python’s import path ● Cleanly isolate application dependencies from platform components ● Dependencies within the environment managed with pip ● Created via: – python3 -m venv (Python 3.4+) – virtualenv (Python 2.7, Python 3.x) ● Not included in the Red Hat Enterprise Linux System Python
  15. 15. Constructing a Layered Application ● Base platform (via system package manager): – Operating system (e.g. kernel, C runtime) – Language runtime (from Software Collections) – Other external dependencies (e.g. OpenSSL) ● Modify shared library loading (via enabled Software Collection) ● Modify Python import configuration (via virtualenv or standard library’s venv) ● Inside the virtual environment: – Application dependencies (managed via pip) – Application source code (managed via pip, or direct from source control)
  16. 16. Deploying a Layered Application (1) ● Application RPM with generated artifacts in SRPM: – Create full installation in representative environment – Bundle entire virtualenv and other desired components into SRPM – Also include scripts to appropriately activate SCL and virtual environment ● Some generated files will end up in the SRPM – Pre-compiled Python files – Executable wrappers for pip managed Python scripts
  17. 17. Deploying a Layered Application (2) ● Application RPM with source-only SRPM: – SRPM contains source for application and any application level dependencies – virtual environment created and configured during RPM build process ● Not yet fully supported in pip – some pip generated metadata will incorrectly include RPM buildroot paths – shouldn’t matter for most RPM based deployment use cases
  18. 18. Managing Application Dependencies ● The Python Package Index is not an App Store! ● Designed to minimize barriers to publication: – No pre-publication review – Publisher Terms of Service ensure right to redistribute, not to run or modify – Publishers may delete (but not replace) previously published versions ● Recommendation: use caching proxies and a component review process – Commercially supported options: JFrog Artifactory, Sonatype Nexus – Community/self-supported options: devpi, Python plugin for Pulp – Check licensing, export restrictions, project governance, ... ● Security monitoring & response also becomes a dev team responsibility!
  19. 19. Python for System Administration
  20. 20. Why *Not* Software Collections? ● Containers are the recommended option for network services ● Software Collections are recommended for applications ● However: – some platform bindings are only installed in the main System Python – the System Python is available on all systems by default ● Some system administration tasks are best handled with the System Python
  21. 21. System Python ● Red Hat Enterprise Linux 6 – Python 2.6.6 (+ selected backports) ● Red Hat Enterprise Linux 7 – Python 2.7.5 (+ selected backports) ● Fedora 23 and later – Recent Python 3.x (rebases rather than backports) – Recent Python 2.7 available as system package, may not have all system bindings
  22. 22. Caveats and Challenges ● Restricted to features of System Python on oldest supported platform ● Community maintained libraries and frameworks often require newer runtimes ● Conflict between supporting: – Red Hat Enterprise Linux 5 (Python 2.4 lacks Py3 forward compatibility features) – Fedora 23+ (Python 3.x as System Python) ● May want to consider higher level system abstractions like Ansible
  23. 23. Migration to Python 3
  24. 24. Python 2.7 Support Timeline ● Anticipated community end-of-life for Python 2.7 in January 2020 – https://docs.python.org/devguide/#status-of-python-branches ● Supported in Red Hat Enterprise Linux 7 until June 2024 – https://access.redhat.com/support/policy/updates/errata ● Anticipate community project support for Python 2 declining sharply post-2020 – Already seeing new community projects starting as Python 3 only
  25. 25. Python 3 Migration Techniques ● General “refactoring enablement” techniques: – automated regression testing frameworks – static structural analysis tools ● Recommended approach for applications and network services: – Migrate to the Python 2.7 SCL or OpenShift image (if using the system Python) – Follow https://docs.python.org/3/howto/pyporting.html – Migrate to the latest Python 3.x SCL or OpenShift image ● Recommended approach for system administration tools: – Consider using Fedora 23+ to look for potential pain points
  26. 26. Python 3 Migration Notes ● Common subset of Python 2.6+ and 3.3+ is quite large ● Many deprecated idioms can be updated automatically ● Key data & workload driven pain points – Explicit bytes/unicode separation – Removal of implicit cross-type comparisons ● Automated refactoring and compatibility testing tools continue to improve
  27. 27. Modernizing the Python 2.7 Network Security Stack
  28. 28. New Security Features in Python 2.7 ● https://docs.python.org/2/whatsnew/2.7.html#pep-466-network-security-enhancements- for-python-2-7 ● Constant-time comparison (hmac.compare_digest()) ● Password storage hashing (hashlib.pbkdf2_hmac()) ● ssl module rebase on Python 3.4 implementation – Server Name Indication support – SSLContext for SSL configuration – Configuration support for TLS 1.x – Access to system certificate stores – ...
  29. 29. Availability in Red Hat Products ● Red Hat Enterprise Linux 7.2 – backported to System Python ● Red Hat Software Collections 2.0+ – default in Python 3.4 ● Red Hat Software Collections 2.2+ – backported to Python 2.7 – default in Python 3.5
  30. 30. Third Party Module Compatibility ● ssl module rebase changed several private implementation details ● Some third party libraries had used internal APIs instead of requesting public ones ● Backport offers greater compatibility than upstream rebase ● Testing before upgrading is still recommended ● Report problems through the usual channels
  31. 31. Defaulting to HTTPS Certificate Verification
  32. 32. What does “HTTPS” Mean? ● Historical Python standard library answer: – “HTTP connection with SSL/TLS enabled” – didn’t check certificate validity or remote host identification ● Modern Python standard library answer: – “What web browsers say it means” – still a HTTP connection with SSL/TLS enabled – also checks for certificate validity – also checks remote host identification against system certificate store
  33. 33. Verifying HTTPS Certificates ● https://docs.python.org/2/whatsnew/2.7.html#pep-476-enabling-certificate-verification-by- default-for-stdlib-http-clients ● Default behavior of standard library HTTPS clients in: – Python 2.7.9+ – Python 3.4.3+ – Python 3.5.0+ ● Turns a silent security failure into a noisy connection failure ● Potential problems: – Self-signed internal certificates – Expired certificates – Internal CAs not configured on client system
  34. 34. On Red Hat Platforms ● File-based opt-in: – Config setting in /etc/python/cert-verification.cfg – Red Hat Enterprise Linux 7.2+ System Python – Red Hat Software Collections 2.2+ Python 2.7 collection ● Default behavior: – Red Hat Software Collections Python 3.5 collection ● Details in Knowledge Base – https://access.redhat.com/articles/2039753
  35. 35. Future Configuration Options ● https://docs.python.org/2/whatsnew/2.7.html#pep-493-https-verification-migration-tools -for-python-2-7 ● Adds new Python 2.7 specific configuration options – ssl._https_verify_certificates() API – PYTHONHTTPSVERIFY environment variable ● Can be used to revert Python 2.7.12+ to Python 2.7.8 behavior ● Python 2.7 only, not supported by any version of Python 3
  36. 36. Resources
  37. 37. OpenShift ● OpenShift – https://www.openshift.com ● OpenShift Online Preview – https://www.openshift.com/devpreview/register.html ● OpenShift Container Development Kit – https://developers.redhat.com/products/cdk/overview/ ● OpenShift Origin – https://www.openshift.org ● OpenShift Origin All-In-One VM – https://www.openshift.org/vm
  38. 38. Python ● Red Hat Software Collections – http://developers.redhat.com/products/softwarecollections/ – Access: https://access.redhat.com/solutions/472793 ● Software Collections Upstream – https://wiki.centos.org/SpecialInterestGroup/SCLo – https://www.softwarecollections.org
  39. 39. Related DevNation 2016 Sessions ● OpenShift – OpenShift Enterprise 3 walk-through with Docker and Kubernetes ● Container Development Kit – CDK 2.0: Docker, Kubernetes, and OSE on your desk – Container development for command line developers ● Software Collections – Software Collections: Easy access to the cutting edge

×