Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SupportingPython3 in Large Scale Project

1,869 views

Published on

(Re-upload in PDF version)

This talk was giving at PyConJP 2019
https://pycon.jp/2019/schedule/

Youtube Video can be found here
https://www.youtube.com/watch?v=BS-HyV3V7GI

Published in: Technology
  • ⇒ www.WritePaper.info ⇐ This service will write as best as they can. So you do not need to waste the time on rewritings.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

SupportingPython3 in Large Scale Project

  1. 1. SupportingPython3 in Large Scale Project Kir Chou A9 (Amazon Product Search) 1
  2. 2. Kir Chou 22 kir.chou note35 2
  3. 3. 3 Outline q Background & Talks q How to support python3? q Execution Plan q Lessons Learned q Things can help you
  4. 4. https://pythonclock.org/ Python 2.7 EOL 2020 Jan 1st 4 Background Python2only = Technical Debt
  5. 5. Talks: Why Python3? • Guido van Rossum (PyCascades 2018) BDFL Python 3 retrospective • Victor Stinner (PyCon 2018) Python 3: ten years later 5
  6. 6. Talks: How to make code Python2/3 compatible? • Ned Batchelder (PyCon2012) Pragmatic Unicode Difference of String in python 2 and 3 • Brett Cannon (PyCon2015) author of SupportingPython3 How to make your code Python 2/3 compatible Strategy to support python3 6
  7. 7. Talks: Learn from others • Jason Fried (PyCon2018) Fighting the Good Fight: Python 3 in your organization Facebook 5 years journey from python2 to 3 • Max Bélanger and Damien DeVille How we rolled out one of the largest Python 3 migrations ever Dropbox journey since 2015 7
  8. 8. How to support python3? 8
  9. 9. 9 Python2 Code Python3 Code Python2/3 Code Python2.x Python3.y Result Ideal world Result
  10. 10. 10 Python2 Code Python3 Code Python2/3 Code Supporting python3 in ideal world – (1) 1. Make all code in python3
  11. 11. 11 Python2 Code Python3 Code Python2/3 Code Python2.x Python3.y Result Supporting python3 in ideal world – (2) Result 2. Make sure the result is same
  12. 12. 12 Python2 Code Python3 Code Python2/3 Code Python2.x Python3.y Supporting python3 in ideal world – (3) Result 3. Stop executing code by python2 interpreter
  13. 13. 13
  14. 14. 真・How to support python3? 14
  15. 15. Large Scale • Packages number: 3 digits • 10+ years old packages • Complicated relationship between packages • Source line of code: Million level • Number of production environment: 2 digits • Build system for python: >1 • Deployment system: >1 15
  16. 16. 16 Large Scale Project @ Real World Python Code PythonX.Y Result Q1. How to find your python code? Q2. What interpreter do your system support? Q3. What’s the intention of the result?
  17. 17. Python Code Q1. How to find your python code? 17 Python Package Non-Python Package Inline Command Company Internal Dependencies External Dependencies
  18. 18. Q2. What interpreter do your system support? • Supported python interpreter? • Default version of the system • Virtualenv • Dockerfile • … • Used python interpreter? • By interpreter (pythonX.Y xxx.py) • By shebang (#!/usr/bin/env python) • By build system to wrap the script • … 18
  19. 19. Q3. What’s the intention of the result? 19 Python Code PythonX.Y Result unit tests site-packages script application Test Runner Build System Deployment System / … Deployment System integration tests Test Server’s Deployment System
  20. 20. Execution Plan 20
  21. 21. 21 1. Prepare POC 2. Build a Team 3. Write Guideline 5. Estimate Task Size 6. Migrate Progressively 4. Investigate Tasks
  22. 22. 1. Prepare PoC Example @ my github 1. Find a tiny migratable package 2. Learn the approach to support python3 3. Prove of concept to support python3 4. Broadcast the idea to other knowledgeable people and human resource allocator 22
  23. 23. 23 Do you need a team for Supporting Python3?
  24. 24. 24 Gathering Knowledges
  25. 25. 25 Gathering Knowledges
  26. 26. 26 Formulating the Guideline Consultation Sharing Knowledges Entry point for external team
  27. 27. 3. Write the Guideline • How to SupportingPython3? (@Background) • How to handle complicated case like “str”? • How to verify the “result”? • Test coverage / Local integration test • Linter (Optional) • Static type checker (Optional) • Integration Test (Not always feasible) • E2E test (Not always feasible) • What’s the supporting python3 strategy? (Explain later) 27
  28. 28. 4. Investigate Tasks (Q1) • Get dependency tree by the Entry-point (Script/Application) 28 Entry-point Dependencies
  29. 29. 4. Investigate Tasks (Q1) 29 🚨 For large scale internal project, there is NO general solution, caniusepython3 can only help external package • We internally developed script to verify that…
  30. 30. 30
  31. 31. 4. Investigate Tasks (Q2 & Q3) • Build System • Assure a reasonable coverage • Assure unit test passed in both versions • Deployment System (In production) • Have site-packages in both versions • Execute part of the script/service in python3 31 PyCon 2019: Thea Flowers (tox, nox)
  32. 32. 5. Estimate Task Size 🚨 It’s hard to measure the task size in general, here are few matrix we used • Original code quality & test coverage • Source lines of code (SLoC) • Internal dependency but own by other team • Edge case (Orphaned open source package) • Build system efforts • Deployment system efforts 32
  33. 33. 6. Migrate Progressively (Transition Period) 33 Transition Period Code is ready in python3, but not yet executed in python3
  34. 34. 6. Migrate Progressively (Progress Tracker) • In the example • Z Dependency is migrated and run in part of production service • Y Dependency is migrated but not yet in any production service • X Application is working in progress. 34 Size Inspected WIP 2/3 Compatible Unit Tested Partially In Prod All In Prod 1 4 5 ② Y Dependency ① Z Dependency ③ X Application
  35. 35. Lessons Learned 35
  36. 36. Always be ready to face Unexpected Chores • Build System / Deployment System • The old system only supports up to python34 ∵ Python34 is EOL, most packages used old system ∴ Work on build system migration in parallel ∵ The old system never run certain package’s unit test ∴ More bugs… • The new build system need workaround to execute linter, type checker in different python version • Many unexpected package without python3 support • 3rd party package – orphaned package • Internal package 36
  37. 37. String The most difficult part of supporting python3 • Not Ascii-code only • Follow the community approach is recommended (100% works) • Example: https://github.com/note35/SupportingPython3-notes • Ascii-code only (Easier) • str can be accepted except some corner cases (~100% works) 37
  38. 38. Things can help you 38
  39. 39. Things can help you • 2 major 3rd party libraries: six and future • Select 1 of them, and add them to the guideline • 2 built-in solution: 2to3 and __future__ • 2to3 only provides limited help • __future__ is a safeguard after supporting python3 • The usage of unicode_literals eventually be dropped 39
  40. 40. Things can help you (cont.) • Linter • pylint or flake8 are recommended • Since you need to touch all legacy packages, it’s more comfortable to read them in unified syntax • Auto-linter: autopep8, black, and yapf • Static Type Checker • mypy is recommended (pyre-check only support 3.5+) • Types helps the progress of code review • By dropbox and our experience 40
  41. 41. Summary 41
  42. 42. If you need to support python3… • SupportPython3 is difficult and long • Side benefits… • Get better understanding of extremely old packages • Make code with higher quality • Ready for next “migration” 42 Time Environment Complexity Project Size
  43. 43. If you do NOT need it? • You know how to estimate the complexity behind your python project now! 43
  44. 44. 44
  45. 45. 45

×