Pycon2017 instagram keynote

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

PYTHON@INSTAGRAM
Hui Ding & Lisa Guo
May 20, 2017

@TechCrunch

STORIES DIRECT LIVE EXPLORE

“INSTAGRAM, WHAT THE HECK
ARE YOU DOING AT PYCON?”
CIRCA PYCON 2015 (MONTREAL)

“HOW DO YOU RUN DJANGO+PYTHON
AT THAT SCALE?”

“HOW DO YOU RUN DJANGO+PYTHON
AT THAT SCALE?”
"WHY HAVEN’T YOU RE-WRITTEN
EVERYTHING IN NODE.JS YET?"

WHY DID INSTAGRAM 
CHOOSE PYTHON?

AND HERE’S WHAT THEY FOUND OUT:
“Friends don’t let friends use RoR”

THINGS INSTAGRAM LOVES ABOUT PYTHON
Easy to become productive
Practicality
Easy to grow 
engineering team
Popular language

THINGS INSTAGRAM LOVES ABOUT DJANGO
Maturity of the
language and
Django framework
Use Django user
model for
supporting 3B+...

WE HAVE TAKEN OUR PYTHON+DJANGO STACK QUITE FAR
1 Sharded database support
2
Run our stack across multiple geographically ...

WE HAVE TAKEN OUR PYTHON+DJANGO STACK QUITE FAR

WE HAVE TAKEN OUR PYTHON+DJANGO STACK QUITE FAR

PYTHON IS SIMPLE AND CLEAN, AND FAVORS PRAGMATISM.
1 Scope the problem, AKA, do simple things ﬁrst
2 Use proven technology...

Subtitle
AT INSTAGRAM, OUR BOTTLENECK IS 
DEVELOPMENT VELOCITY,  
NOT PURE CODE EXECUTION
BUT PYTHON IS STILL SLOW, RIGHT?

SCALING PYTHON TO SUPPORT USER AND FEATURE GROWTH
20
40
60
80
00
0 2 4 6 8 10 12 14 16 18 20 22 24Server growthUser growth

PYTHON EFFICIENCY STRATEGY
1 Build extensive tools to proﬁle and understand perf bottleneck
2 Moving stable, critical comp...

ROAD TO PYTHON 3

1 Motivation
2 Strategy
3 Challenges
4 Resolution

MOTIVATION

def compose_from_max_id(max_id):
‘’’ @param str max_id ’’’
MOTIVATION: DEV VELOCITY

MOTIVATION: PERFORMANCE
uWSGI/web async tier/celery
media storage
user/media
metadata
search/ranking
Python

MOTIVATION: PERFORMANCE
N processes
M CPU cores
N >> M
Request

MOTIVATION: PERFORMANCE
N processes
M CPU cores
N == M
Request

MOTIVATION: COMMUNITY

STRATEGIES

SERVICE DOWNTIME

SERVICE DOWNTIME
PRODUCT SLOWDOWN

MASTER
LIVE
DEVELOP/TEST/DOGFOOD

Create separate branch?
MIGRATION OPTIONS
• Branch sync overhead, error prone
• Merging back will be a risk
• Lose the opp...

MIGRATION OPTIONS
One endpoint at a time?Create separate branch?
• Common modules across end points
• Context switch for d...

MIGRATION OPTIONS
One endpoint at a time?Create separate branch? Micro services?
• Massive code restructuring
• Higher lat...

MIGRATION OPTIONS
One endpoint at a time?Create separate branch? Micro services?

MAKE MASTER COMPATIBLE

MASTER
PYTHON3
PYTHON2

Third-party packages 
3-4 months
Codemod 
2-3 months
Unit tests 
2 months
Production rollout 
4 months

THIRD-PARTY PACKAGES
Rule: No Python3, no new package

Rule: No Python3, no new package
Delete unused, incompatible packages
twisted
django-paging
django-sentry
django-templatet...

Upgraded packages
Rule: No Python3, no new package
Delete unused, incompatible packages
THIRD-PARTY PACKAGES

CODEMOD
modernize -f libmodernize.fixes.fix_filter <dir> -w -n
raise, metaclass, methodattr, next, funcattr, library renam...

Failed
include_list: passed tests
Passed
UNIT TESTS

FailedPassed
UNIT TESTS

exclude_list: failed tests
FailedPassed
UNIT TESTS

• Not 100% code coverage
• Many external services have mocks
• Data compatibility issues typically do not show up in unit ...

100%
20%
0.1%
EMPLOYEES
DEVELOPERS
ROLLOUT

100%
20%
0.1%
EMPLOYEES
DEVELOPERS
ROLLOUT

CHALLENGES

CHALLENGES
1 Unicode
2 Data format incompatible
3
4 Dictionary ordering
Iterator

from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future...

from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future...

CHALLENGE: UNICODE/STR/BYTES
mymac = hmac.new(‘abc’)
TypeError: key: expected bytes or bytearray, but got 'str'
value = ‘a...

CHALLENGE: UNICODE/STR/BYTES
ensure_binary()
ensure_str()
ensure_text()
mymac = hmac.new(ensure_binary(‘abc’))
Helper func...

CHALLENGE: PICKLE
memcache_data = pickle.dumps(data, pickle.HIGHEST_PROTOCOL)
data = pickle.load(memcache_data)
Write
Read...

CHALLENGE: PICKLE
Memcache
Me
Python3
Others
Python2
memcache_data = pickle.dumps(data, pickle.HIGHEST_PROTOCOL)
data = pi...

CHALLENGE: PICKLE
pickle.dumps({'a': '爱'}, 2) UnicodeDecodeError: 'ascii' codec
can’t decode byte 0xe9 in position
0: ordi...

$pickle.dumps({'a': '爱'}, 2){u'a': u'u7231'} != {'a': '爱'} Python2 reads Python3 writes memcache_data = pickle.dumps(data, ...$

CHALLENGE: PICKLE
Memcache
Python3
Python2
4
Memcache
2

map()
filter()
dict.items()

1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for bui...

1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for bui...

1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for bui...

1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = list(map(BuildProcess, CYTHON_SOURCES))
3 while any(not build.done() f...

'{"a": 1, "c": 3, "b": 2}'
CHALLENGE: DICTIONARY ORDER
Python2
Python3.5.1
>>> testdict = {'a': 1, 'b': 2, 'c': 3}
>>> jso...

ALMOST THERE...

CPU instructions per request
max Requests Per Second
-12%
0%
Memory conﬁguration diﬀerence?
12%

if uwsgi.opt.get('optimize_mem', None) == 'True':
optimize_mem()
b

RESOLUTION

FEB 2017
PYTHON3
PYTHON2

Saving of 30% 
(on celery)
INSTAGRAM ON PYTHON3
Saving of 12% 
(on uwsgi/django)
CPU MEMORY

June 2016
Python3 migration
Sept 2015 Dec 2016 Apr 2017
400M
500M 600M

Video View Notiﬁcation
Save Draft
Comment Filtering
Story Viewer Ranking
First Story Notiﬁcation
Self-harm Prevention
Live

MOTIVATION: TYPE HINTS
Type hints - 2% done
def compose_from_max_id(max_id: Optional[str]) -> Optional[str]:
MyPy and type...

MOTIVATION: ASYNC IO
Asynchronize web framework
Parallel network access within a request

MOTIVATION: COMMUNITY
Benchmark web workload
Run time, memory proﬁling, etc

Pycon2017 instagram keynote

Pycon2017 instagram keynote

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1045526 views
Storytelling For The Web: Integrate... by Chiara Aliotta 958884 views
Artificial Intelligence, Data and C... by OECD Directorate ... 886438 views
How to Leverage AI to Boost Employe... by SocialHRCamp 373696 views
2024 State of Marketing Report – by... by Marius Sescu 212106 views
Everything You Need To Know About C... by Expeed Software 231238 views

View on Slideshare

1 of 85 Ad

View on Slideshare

1 of 85 Ad