1. PYTHON@INSTAGRAM
Hui Ding & Lisa Guo
May 20, 2017

3. @TechCrunch

5. STORIES DIRECT LIVE EXPLORE

6. “INSTAGRAM, WHAT THE HECK
ARE YOU DOING AT PYCON?”
CIRCA PYCON 2015 (MONTREAL)

7. “HOW DO YOU RUN DJANGO+PYTHON
AT THAT SCALE?”

8. “HOW DO YOU RUN DJANGO+PYTHON
AT THAT SCALE?”
"WHY HAVEN’T YOU RE-WRITTEN
EVERYTHING IN NODE.JS YET?"

9. WHY DID INSTAGRAM
CHOOSE PYTHON?

10. AND HERE’S WHAT THEY FOUND OUT:
“Friends don’t let friends use RoR”

11. THINGS INSTAGRAM LOVES ABOUT PYTHON
Easy to become productive
Practicality
Easy to grow
engineering team
Popular language

12. THINGS INSTAGRAM LOVES ABOUT DJANGO
Maturity of the
language and
Django framework
Use Django user
model for
supporting 3B+
registered users

13. WE HAVE TAKEN OUR PYTHON+DJANGO STACK QUITE FAR
1 Sharded database support
2
Run our stack across multiple geographically distributed data centers3
Disable garbage-collection to improve memory utilization

14. WE HAVE TAKEN OUR PYTHON+DJANGO STACK QUITE FAR

15. WE HAVE TAKEN OUR PYTHON+DJANGO STACK QUITE FAR

16. PYTHON IS SIMPLE AND CLEAN, AND FAVORS PRAGMATISM.
1 Scope the problem, AKA, do simple things first
2 Use proven technology
3 User first: focus on adding value to user facing features

17. Subtitle
AT INSTAGRAM, OUR BOTTLENECK IS
DEVELOPMENT VELOCITY,
NOT PURE CODE EXECUTION
BUT PYTHON IS STILL SLOW, RIGHT?

19. SCALING PYTHON TO SUPPORT USER AND FEATURE GROWTH
20
40
60
80
00
0 2 4 6 8 10 12 14 16 18 20 22 24Server growthUser growth

20. PYTHON EFFICIENCY STRATEGY
1 Build extensive tools to profile and understand perf bottleneck
2 Moving stable, critical components to C/C++, e.g., memcached access
3
Async? New python runtime?4
Cythonization

22. ROAD TO PYTHON 3

23. 1 Motivation
2 Strategy
3 Challenges
4 Resolution

24. MOTIVATION

26. def compose_from_max_id(max_id):
‘’’ @param str max_id ’’’
MOTIVATION: DEV VELOCITY

27. MOTIVATION: PERFORMANCE
uWSGI/web async tier/celery
media storage
user/media
metadata
search/ranking
Python

28. MOTIVATION: PERFORMANCE
N processes
M CPU cores
N >> M
Request

29. MOTIVATION: PERFORMANCE
N processes
M CPU cores
N == M
Request

30. MOTIVATION: COMMUNITY

31. STRATEGIES

32. SERVICE DOWNTIME

33. SERVICE DOWNTIME
PRODUCT SLOWDOWN

34. MASTER
LIVE
DEVELOP/TEST/DOGFOOD

35. Create separate branch?
MIGRATION OPTIONS
• Branch sync overhead, error prone
• Merging back will be a risk
• Lose the opportunity to educate

36. MIGRATION OPTIONS
One endpoint at a time?Create separate branch?
• Common modules across end points
• Context switch for developers
• Overhead of managing separate pools

37. MIGRATION OPTIONS
One endpoint at a time?Create separate branch? Micro services?
• Massive code restructuring
• Higher latency
• Deployment complexity

38. MIGRATION OPTIONS
One endpoint at a time?Create separate branch? Micro services?

39. MAKE MASTER COMPATIBLE

40. MASTER
PYTHON3
PYTHON2

41. Third-party packages
3-4 months
Codemod
2-3 months
Unit tests
2 months
Production rollout
4 months

42. THIRD-PARTY PACKAGES
Rule: No Python3, no new package

43. Rule: No Python3, no new package
Delete unused, incompatible packages
twisted
django-paging
django-sentry
django-templatetag-sugar
dnspython
enum34
hiredis
httplib2
ipaddr
jsonfig
pyapns
phpserialize
python-memcached
thrift
THIRD-PARTY PACKAGES

44. Upgraded packages
Rule: No Python3, no new package
Delete unused, incompatible packages
THIRD-PARTY PACKAGES

45. CODEMOD
modernize -f libmodernize.fixes.fix_filter <dir> -w -n
raise, metaclass, methodattr, next, funcattr, library renaming, range,
maxint/maxsize, filter->list comprehension, long integer, itertools,
tuple unpacking in method, cython, urllib parse, StringiO, context
mgr/decorator, ipaddr, cmp, except, nested, dict iter fixes, mock

47. Failed
include_list: passed tests
Passed
UNIT TESTS

48. FailedPassed
UNIT TESTS

49. exclude_list: failed tests
FailedPassed
UNIT TESTS

52. • Not 100% code coverage
• Many external services have mocks
• Data compatibility issues typically do not show up in unit tests
UNIT TESTS: LIMITS

53. 100%
20%
0.1%
EMPLOYEES
DEVELOPERS
ROLLOUT

54. 100%
20%
0.1%
EMPLOYEES
DEVELOPERS
ROLLOUT

55. CHALLENGES

56. CHALLENGES
1 Unicode
2 Data format incompatible
3
4 Dictionary ordering
Iterator

57. from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
CHALLENGE: UNICODE/STR/BYTES

58. from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals
CHALLENGE: UNICODE/STR/BYTES
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals

59. CHALLENGE: UNICODE/STR/BYTES
mymac = hmac.new(‘abc’)
TypeError: key: expected bytes or bytearray, but got 'str'
value = ‘abc’
if isinstance(value, six.text_type):
value = value.encode(encoding=‘utf-8’)
mymac = hmac.new(value)
Error
Fix

60. CHALLENGE: UNICODE/STR/BYTES
ensure_binary()
ensure_str()
ensure_text()
mymac = hmac.new(ensure_binary(‘abc’))
Helper functions
Fix

61. CHALLENGE: PICKLE
memcache_data = pickle.dumps(data, pickle.HIGHEST_PROTOCOL)
data = pickle.load(memcache_data)
Write
Read
Memcache
Me
Python3
Others
Python2

62. CHALLENGE: PICKLE
Memcache
Me
Python3
Others
Python2
memcache_data = pickle.dumps(data, pickle.HIGHEST_PROTOCOL)
data = pickle.load(memcache_data)
Write
Read
4
ValueError: unsupported pickle protocol: 4
2

63. CHALLENGE: PICKLE
pickle.dumps({'a': '爱'}, 2) UnicodeDecodeError: 'ascii' codec
can’t decode byte 0xe9 in position
0: ordinal not in range(128)
memcache_data = pickle.dumps(data, 2)
Write
Python2 writes Python3 reads

64. pickle.dumps({'a': '爱'}, 2){u'a': u'u7231'} != {'a': '爱'}
Python2 reads Python3 writes
memcache_data = pickle.dumps(data, 2)
Write
CHALLENGE: PICKLE

65. CHALLENGE: PICKLE
Memcache
Python3
Python2
4
Memcache
2

67. map()
filter()
dict.items()

68. 1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR

69. 1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR

70. 1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR

71. 1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = list(map(BuildProcess, CYTHON_SOURCES))
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR

72. '{"a": 1, "c": 3, "b": 2}'
CHALLENGE: DICTIONARY ORDER
Python2
Python3.5.1
>>> testdict = {'a': 1, 'b': 2, 'c': 3}
>>> json.dumps(testdict)
Python3.6
Cross version
'{"c": 3, "b": 2, "a": 1}'
'{"c": 3, "a": 1, "b": 2}'
'{"a": 1, "b": 2, "c": 3}'
>>> json.dumps(testdict, sort_keys=True)
'{"a": 1, "b": 2, "c": 3}'

73. ALMOST THERE...

74. CPU instructions per request
max Requests Per Second
-12%
0%
Memory configuration difference?
12%

75. if uwsgi.opt.get('optimize_mem', None) == 'True':
optimize_mem()
b

76. RESOLUTION

77. FEB 2017
PYTHON3
PYTHON2

78. Saving of 30%
(on celery)
INSTAGRAM ON PYTHON3
Saving of 12%
(on uwsgi/django)
CPU MEMORY

79. June 2016
Python3 migration
Sept 2015 Dec 2016 Apr 2017
400M
500M 600M

80. Video View Notification
Save Draft
Comment Filtering
Story Viewer Ranking
First Story Notification
Self-harm Prevention
Live

81. MOTIVATION: TYPE HINTS
Type hints - 2% done
def compose_from_max_id(max_id: Optional[str]) -> Optional[str]:
MyPy and typeshed contribution
Tooling - collect data and suggest type hints

82. MOTIVATION: ASYNC IO
Asynchronize web framework
Parallel network access within a request

83. MOTIVATION: COMMUNITY
Benchmark web workload
Run time, memory profiling, etc