It's 2019 & I'm still using Python 2! Should I be worried?

Witowski Sebastian
ヴィトヴスキ・セバスティアン
IT’S 2019
&
I’M STILL USING PYTHON 2!
SHOULD I BE WORRIED?

@SebaWitowski bit.ly/it-is-2019
Technical remarks
Slides: bit.ly/it-is-2019
@SebaWitowski
https://switowski.com

$whoami
Python consultant and trainer
@SebaWitowski
https://switowski.com/blog

Who is still using Python 2?
(hands up)

Who is planning to migrate
to Python 3 before the end
of the year?

Who will still use Python 2
in 2020?

About this talk…

It’s not “How we migrated to
Python 3 at FooBar company”
• Migrating Pinterest from Python2 to Python3 - PyCon 2019
• Instagram Makes a Smooth Move to Python 3
• How Dropbox migrated to Python 3
• Python 2 is dead! Drag your old code into the modern age

It does not focus on
the migration part

We will talk about
• Risks of staying on an unsupported Python version
• Different possible solutions for your project
• And finally - the migration itself

Python 3 has been out
for over 10 years
(03.12.2008)

Why Python 2 is still used?
• The cost of migration is too high from a
business point of view

“Rewriting Python 2 to
Python 3 does not bring
money for the company!”

• There is always a new feature or an urgent fix

“2 things that managers (usually)
won’t agree to do:
1. Spend time on refactoring
2. Rewrite a working application”

Refactoring can be done in
small increments (following the
“boy scout” rule)

Migration to Python 3 can’t*

• There is always a new feature or an urgent fix
• Migrating an old codebase can be scary

“Migration to Python 3
does not bring any value?”

Benefits of migrating to Python 3
• You won’t be using an unsupported version of
a programming language in a few months

• Python 3 is faster than Python 2 …

• … and it has new features (asyncio, type hints,
ordered dictionaries, f-strings, better unicode)!

• Your dependencies are already on Python 3 …

• … and they will stop supporting Python 2 soon!

https://python3statement.org/

• … and they will stop supporting Python 2 soon!
Show this slide to your manager!

Now is the best time to migrate!

• You dependencies are probably already migrated
(which was not the case a few years ago)

@SebaWitowski bit.ly/it-is-2019http://py3readiness.org/

• Your dependencies are probably already migrated

• Your dependencies are probably already migrated
• The automatic tools have been tested by
thousands of developers

• The automatic tools have been tested by
thousands of developers
• You can learn from others …

• You can learn from others:
• https://blogs.dropbox.com/tech/2019/02/
incrementally-migrating-over-one-million-lines-
of-code-from-python-2-to-python-3/
• https://docs.python.org/3/howto/pyporting.html
• https://www.youtube.com/watch?
v=66XoCk79kjM

Lisa Guo, Hui Ding Keynote PyCon 2017

• The automatic tools has been tested by thousands
of developers
• You can learn from others
• And you still have those 2-3 months of “runway”

Python 2 EOL
https://pythonclock.org/ a.k.a. Python death clock

Bitter Sweet
• First
• Second
• Third
• Fourth

No more bug fixes

A 0-day exploit for Python2?
Good Luck, Have Fun fixing that!

Commercial vendors
You won’t get support from the Python community,
but you might get it from commercial vendors!
E.g. RHEL7 that was released in 2014 is still using
Python 2.7, so Red Hat should provide you support
until 2024 (based on their 10-year support lifecycle).
If you are willing to pay for it.

What will happen with your app?

• Nothing

• Nothing
• It will be hacked

• Nothing
• It will be hacked
• It will slowly fall apart

What can you do about Python 2 EOL?

• Do nothing
• Freeze the state of your application (docker)
• Change Python interpreter
• Maintain your own Python 2 build
• Migrate to Python 3
• Rewrite you application
Low
High
Effort

Freeze the state of your application
People can remove packages from PyPI or GitHub!

“How badly are you screwed if
one of your dependencies
disappears RIGHT NOW?”
Ask yourself:

• Write a Dockerfile that uses Python 2 as the base image
• Add all dependencies
• Set your app as a Docker image
• Push that image to a public/private docker repository
• Now you have an immutable container with your app!

• It’s a good approach for internal tools
• Please do this now! Not in 2020!

Change Python interpreter
PyPy - a good alternative to
CPython that is not planning
to deprecate Python 2

PyPy is very mature and companies are using it in production
Using PyPy instead of Python for speed by Niklas Bivald

PyPy will probably not work with C extensions
(but it will with cffi and ctypes!)

https://software.intel.com/en-us/distribution-for-python

Commercial solutions
(RHEL 7)
Disclaimer: I’m not related with any company that I’m mentioning in this talk!

• Do nothing
• Maintain your own CPython 2 build
• Rewrite you application
Low
High
Effort

Migrate to Python 3

Migrate to Python 3
• With straddling code (code that supports
both Python 2 and Python 3)
• Run tests under Python 3 (they will fail)
• Rewrite your application until it works
under Python 2 and 3
• Switch Python version in production
• [OPTIONAL] Remove the Python 2 code

Migrate to Python 3
• With straddling code (code that supports
both Python 2 and Python 3)
• Rewriting Python 2 code to Python 3
• Less work (you don’t care about Python 2)
• Scary to switch (what if you didn’t test
something?)
• Requires feature freeze

• Do nothing
• Maintain your own CPython 2 build
• Rewrite your application
Low
High
Effort

Agenda
• Risks of staying on an unsupported Python version
• Different possible solutions for your project
• And finally - the migration itself

How to prepare for the migration?

Migration plan
1. You need tests
2. Seriously, you NEED tests!
3. Check dependencies
4. Migrate

Migration plan
1. You need tests
2. Seriously, you NEED tests!
4. Migrate
It applies also to PyPy / commercial distribution migration

1. You need tests
If there is technical debt, two things suffer first:
1. Tests
2. Documentation

1. You need tests
Many projects are not yet migrated to Python 3
because they lack tests.

“Hey boss, before we spend those 3 months
migrating our project to Python 3, we need to
spend 2 more weeks writing tests in Python 2.
We will have to migrate those tests as well.”

1. You need tests
1. Write your tests so they work with Python 2 and 3

1. You need tests
1. Write your tests so they work with Python 2 and 3
2. If you absolutely can’t afford to write tests and your
project is a web application - record functional
tests

https://www.seleniumhq.org/selenium-ide/

Record your tests in Selenium
• Record every possible use case in your application:
• Click every button
• Fill in every form
• Make mistakes to see error messages
• Use assertions to check if data was preserved
correctly (string encoding will be one of the main
problems during the migration)!

Back up those tests!

Profit
• You just created a decent test coverage in 2 days of
work:
• No backend tests
• Slow to run
• But they test what your customers see!

Demo

Puppeteer

Puppeteer Recorder

Paid solutions

2. You need tests
If you have time to actually write tests focus on:
1. Unicode support (test areas where the strings are
getting IN and OUT of your application)
2. Division (test functions that are dividing numbers -
especially when dealing with money!)

Set up tox
https://tox.readthedocs.io

Set up tox
• Install: pip install tox
• Create tox.ini file:
• Make sure you have Python 2.7 and 3.7 installed!
• Run tox command
# content of: tox.ini , put in same dir as setup.py
[tox]
envlist = py27,py37
skipsdist = True
[testenv]
# install pytest in the virtualenv where commands will be executed
deps = pytest
commands =
# NOTE: you can run any command line tool here - not just tests
python -m pytest

Set up tox

• The most popular dependencies are already migrated
• Use the caniusepython3 tool

caniusepython3
https://github.com/brettcannon/caniusepython3

caniusepython3
caniusepython3 checks the metadata of each package!
If the metadata is incorrect (wrong trove classifiers), it
might return wrong information!

caniusepython3 in practice
Looks like mechanize and nydus packages
don’t have Python 3 versions yet!
Let’s check the mechanize…

https://github.com/python-mechanize/mechanize

https://pypi.org/project/MechanicalSoup/

4. Migrate
1. Use automatic tools to do most of the work
2. Fix the rest manually

Automatic tools
They can do most of the repetitive work
(replace print statement with function, change
modules to their Python 3 equivalents, etc.).
But their output has to be checked by you!

2to3
https://docs.python.org/2/library/2to3.html

Futurize and Modernize
https://python-modernize.readthedocs.io/en/latest/index.htmlhttps://python-future.org/futurize.html

https://python-future.org/futurize.html
Futurize in practice

lib2to3.fixes.fix_apply
lib2to3.fixes.fix_except
lib2to3.fixes.fix_exitfunc
lib2to3.fixes.fix_funcattrs
lib2to3.fixes.fix_has_key
lib2to3.fixes.fix_idioms
lib2to3.fixes.fix_intern
lib2to3.fixes.fix_isinstance
lib2to3.fixes.fix_methodattrs
lib2to3.fixes.fix_ne
lib2to3.fixes.fix_numliterals
lib2to3.fixes.fix_paren
lib2to3.fixes.fix_reduce
lib2to3.fixes.fix_renames
lib2to3.fixes.fix_repr
lib2to3.fixes.fix_standarderror
lib2to3.fixes.fix_sys_exc
lib2to3.fixes.fix_throw
lib2to3.fixes.fix_tuple_params
lib2to3.fixes.fix_types
lib2to3.fixes.fix_ws_comma
lib2to3.fixes.fix_xreadlines
libfuturize.fixes.fix_absolute_import
libfuturize.fixes.fix_next_call
libfuturize.fixes.fix_print_with_import
libfuturize.fixes.fix_raise
lib2to3.fixes.fix_basestring
lib2to3.fixes.fix_dict
lib2to3.fixes.fix_exec
lib2to3.fixes.fix_getcwdu
lib2to3.fixes.fix_input
lib2to3.fixes.fix_itertools
lib2to3.fixes.fix_itertools_imports
lib2to3.fixes.fix_filter
lib2to3.fixes.fix_long
lib2to3.fixes.fix_map
lib2to3.fixes.fix_nonzero
lib2to3.fixes.fix_operator
lib2to3.fixes.fix_raw_input
lib2to3.fixes.fix_zip
libfuturize.fixes.fix_cmp
libfuturize.fixes.fix_division
libfuturize.fixes.fix_execfile
libfuturize.fixes.fix_future_builtins
libfuturize.fixes.fix_future_standard_library
libfuturize.fixes.fix_future_standard_library_urllib
libfuturize.fixes.fix_metaclass
libpasteurize.fixes.fix_newstyle
libfuturize.fixes.fix_object
libfuturize.fixes.fix_unicode_keep_u
libfuturize.fixes.fix_xrange_with_import

Run a single fix:
$ futurize -f has_key .
$ futurize -w -f has_key .

Fixes are divided into 2 categories:
• Stage 1 - “safe” modifications (don’t break Python 2.6
compatibility, don’t introduce new dependencies)
• Stage 2 - others (introduce dependencies from future
package, rename standard library imports to Python 3
versions, etc.)

You can run the whole stages of fixes at once:
• futurize --stage1
• futurize --stage2
• But it’s better to run fixes one by one
• Don’t mix automatic fixes with manual changes!

“But that’s 53 fixes that I have to run!
It’s boring!”
https://github.com/tdhopper/auto-futurize

Next, you have to decide what to do with strings:
A good explanation of Unicode problems and how to
solve them:
https://nedbatchelder.com/text/unipain.html
from builtins import bytes, str
b = bytes(b'x00ABCD')
s = str(u'This is normal text')

A sane approach to Unicode
“You should have bytes on the outside of your system
and Unicode on the inside.
You should perform encode/decode on the edges of
your system.”

4. Migrate
<<<WE ARE HERE>>>

4. Migrate
🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳 🥳

Some good resources that can help you:
• http://python3porting.com/toc.html
• http://blog.pyspoken.com/2018/02/13/python-2-
to-3-migration-guide/
• https://portingguide.readthedocs.io
• YouTube videos from conferences
More resources:

Conclusions
• You don’t have to migrate to Python 3
• You can Dockerize your Python 2 application
• You can use PyPy
• Or a commercial vendor

Conclusions
• But now is the best time to migrate:
• Your dependencies are already migrated
• Migration tools are mature and well tested
• Python 2 is still supported for a while

Conclusions
So convince your manager that the best feature
to add to your project is Python 3 support 😉
And if you need help - I’m available to hire!

THANK
YOU

It's 2019 & I'm still using Python 2! Should I be worried?

More Related Content

Similar to It's 2019 & I'm still using Python 2! Should I be worried?

More from Sebastian Witowski

Recently uploaded

It's 2019 & I'm still using Python 2! Should I be worried?