Downloaded 28 times
![1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR](https://image.slidesharecdn.com/pycon2017instagramkeynote-170523221745/75/Pycon2017-instagram-keynote-68-2048.jpg)
![1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR](https://image.slidesharecdn.com/pycon2017instagramkeynote-170523221745/75/Pycon2017-instagram-keynote-69-2048.jpg)
![1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = map(BuildProcess, CYTHON_SOURCES)
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR](https://image.slidesharecdn.com/pycon2017instagramkeynote-170523221745/75/Pycon2017-instagram-keynote-70-2048.jpg)
![1 CYTHON_SOURCES = [a.pyx, b.pyx, c.pyx]
2 builds = list(map(BuildProcess, CYTHON_SOURCES))
3 while any(not build.done() for build in builds):
4 pending = [build for build in builds if not build.started()]
<do some work>
CHALLENGE: ITERATOR](https://image.slidesharecdn.com/pycon2017instagramkeynote-170523221745/75/Pycon2017-instagram-keynote-71-2048.jpg)
![MOTIVATION: TYPE HINTS
Type hints - 2% done
def compose_from_max_id(max_id: Optional[str]) -> Optional[str]:
MyPy and typeshed contribution
Tooling - collect data and suggest type hints](https://image.slidesharecdn.com/pycon2017instagramkeynote-170523221745/75/Pycon2017-instagram-keynote-81-2048.jpg)
More Related Content
Similar to Pycon2017 instagram keynote
Pycon2017 instagram keynote
- 1. PYTHON@INSTAGRAM Hui Ding &Lisa Guo May 20, 2017
- 3.
- 5. STORIES DIRECT LIVEEXPLORE
- 6. “INSTAGRAM, WHAT THEHECK ARE YOU DOING AT PYCON?” CIRCA PYCON 2015 (MONTREAL)
- 7. “HOW DO YOURUN DJANGO+PYTHON AT THAT SCALE?”
- 8. “HOW DO YOURUN DJANGO+PYTHON AT THAT SCALE?” "WHY HAVEN’T YOU RE-WRITTEN EVERYTHING IN NODE.JS YET?"
- 9. WHY DID INSTAGRAM CHOOSEPYTHON?
- 10. AND HERE’S WHATTHEY FOUND OUT: “Friends don’t let friends use RoR”
- 11. THINGS INSTAGRAM LOVESABOUT PYTHON Easy to become productive Practicality Easy to grow engineering team Popular language
- 12. THINGS INSTAGRAM LOVESABOUT DJANGO Maturity of the language and Django framework Use Django user model for supporting 3B+ registered users
- 13. WE HAVE TAKENOUR PYTHON+DJANGO STACK QUITE FAR 1 Sharded database support 2 Run our stack across multiple geographically distributed data centers3 Disable garbage-collection to improve memory utilization
- 14. WE HAVE TAKENOUR PYTHON+DJANGO STACK QUITE FAR
- 15. WE HAVE TAKENOUR PYTHON+DJANGO STACK QUITE FAR
- 16. PYTHON IS SIMPLEAND CLEAN, AND FAVORS PRAGMATISM. 1 Scope the problem, AKA, do simple things first 2 Use proven technology 3 User first: focus on adding value to user facing features
- 17. Subtitle AT INSTAGRAM, OURBOTTLENECK IS DEVELOPMENT VELOCITY, NOT PURE CODE EXECUTION BUT PYTHON IS STILL SLOW, RIGHT?
- 19. SCALING PYTHON TOSUPPORT USER AND FEATURE GROWTH 20 40 60 80 00 0 2 4 6 8 10 12 14 16 18 20 22 24Server growthUser growth
- 20. PYTHON EFFICIENCY STRATEGY 1Build extensive tools to profile and understand perf bottleneck 2 Moving stable, critical components to C/C++, e.g., memcached access 3 Async? New python runtime?4 Cythonization
- 22.
- 23. 1 Motivation 2 Strategy 3Challenges 4 Resolution
- 24.
- 26. def compose_from_max_id(max_id): ‘’’ @paramstr max_id ’’’ MOTIVATION: DEV VELOCITY
- 27. MOTIVATION: PERFORMANCE uWSGI/web asynctier/celery media storage user/media metadata search/ranking Python
- 28. MOTIVATION: PERFORMANCE N processes MCPU cores N >> M Request
- 29. MOTIVATION: PERFORMANCE N processes MCPU cores N == M Request
- 30.
- 31.
- 32.
- 33.
- 34.
- 35. Create separate branch? MIGRATIONOPTIONS • Branch sync overhead, error prone • Merging back will be a risk • Lose the opportunity to educate
- 36. MIGRATION OPTIONS One endpointat a time?Create separate branch? • Common modules across end points • Context switch for developers • Overhead of managing separate pools
- 37. MIGRATION OPTIONS One endpointat a time?Create separate branch? Micro services? • Massive code restructuring • Higher latency • Deployment complexity
- 38. MIGRATION OPTIONS One endpointat a time?Create separate branch? Micro services?
- 39.
- 40.
- 41. Third-party packages 3-4 months Codemod 2-3months Unit tests 2 months Production rollout 4 months
- 42. THIRD-PARTY PACKAGES Rule: NoPython3, no new package
- 43. Rule: No Python3,no new package Delete unused, incompatible packages twisted django-paging django-sentry django-templatetag-sugar dnspython enum34 hiredis httplib2 ipaddr jsonfig pyapns phpserialize python-memcached thrift THIRD-PARTY PACKAGES
- 44. Upgraded packages Rule: NoPython3, no new package Delete unused, incompatible packages THIRD-PARTY PACKAGES
- 45. CODEMOD modernize -f libmodernize.fixes.fix_filter<dir> -w -n raise, metaclass, methodattr, next, funcattr, library renaming, range, maxint/maxsize, filter->list comprehension, long integer, itertools, tuple unpacking in method, cython, urllib parse, StringiO, context mgr/decorator, ipaddr, cmp, except, nested, dict iter fixes, mock
- 47.
- 48.
- 49.
- 52. • Not 100%code coverage • Many external services have mocks • Data compatibility issues typically do not show up in unit tests UNIT TESTS: LIMITS
- 53.
- 54.
- 55.
- 56. CHALLENGES 1 Unicode 2 Dataformat incompatible 3 4 Dictionary ordering Iterator
- 57. from __future__ importabsolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals CHALLENGE: UNICODE/STR/BYTES
- 58. from __future__ importabsolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals CHALLENGE: UNICODE/STR/BYTES from __future__ import absolute_import from __future__ import print_function from __future__ import division from __future__ import unicode_literals
- 59. CHALLENGE: UNICODE/STR/BYTES mymac =hmac.new(‘abc’) TypeError: key: expected bytes or bytearray, but got 'str' value = ‘abc’ if isinstance(value, six.text_type): value = value.encode(encoding=‘utf-8’) mymac = hmac.new(value) Error Fix
- 60. CHALLENGE: UNICODE/STR/BYTES ensure_binary() ensure_str() ensure_text() mymac =hmac.new(ensure_binary(‘abc’)) Helper functions Fix
- 61. CHALLENGE: PICKLE memcache_data =pickle.dumps(data, pickle.HIGHEST_PROTOCOL) data = pickle.load(memcache_data) Write Read Memcache Me Python3 Others Python2
- 62. CHALLENGE: PICKLE Memcache Me Python3 Others Python2 memcache_data =pickle.dumps(data, pickle.HIGHEST_PROTOCOL) data = pickle.load(memcache_data) Write Read 4 ValueError: unsupported pickle protocol: 4 2
- 63. CHALLENGE: PICKLE pickle.dumps({'a': '爱'},2) UnicodeDecodeError: 'ascii' codec can’t decode byte 0xe9 in position 0: ordinal not in range(128) memcache_data = pickle.dumps(data, 2) Write Python2 writes Python3 reads
- 64. pickle.dumps({'a': '爱'}, 2){u'a':u'u7231'} != {'a': '爱'} Python2 reads Python3 writes memcache_data = pickle.dumps(data, 2) Write CHALLENGE: PICKLE
- 65.
- 67.
- 68. 1 CYTHON_SOURCES =[a.pyx, b.pyx, c.pyx] 2 builds = map(BuildProcess, CYTHON_SOURCES) 3 while any(not build.done() for build in builds): 4 pending = [build for build in builds if not build.started()] <do some work> CHALLENGE: ITERATOR
- 69. 1 CYTHON_SOURCES =[a.pyx, b.pyx, c.pyx] 2 builds = map(BuildProcess, CYTHON_SOURCES) 3 while any(not build.done() for build in builds): 4 pending = [build for build in builds if not build.started()] <do some work> CHALLENGE: ITERATOR
- 70. 1 CYTHON_SOURCES =[a.pyx, b.pyx, c.pyx] 2 builds = map(BuildProcess, CYTHON_SOURCES) 3 while any(not build.done() for build in builds): 4 pending = [build for build in builds if not build.started()] <do some work> CHALLENGE: ITERATOR
- 71. 1 CYTHON_SOURCES =[a.pyx, b.pyx, c.pyx] 2 builds = list(map(BuildProcess, CYTHON_SOURCES)) 3 while any(not build.done() for build in builds): 4 pending = [build for build in builds if not build.started()] <do some work> CHALLENGE: ITERATOR
- 72. '{"a": 1, "c":3, "b": 2}' CHALLENGE: DICTIONARY ORDER Python2 Python3.5.1 >>> testdict = {'a': 1, 'b': 2, 'c': 3} >>> json.dumps(testdict) Python3.6 Cross version '{"c": 3, "b": 2, "a": 1}' '{"c": 3, "a": 1, "b": 2}' '{"a": 1, "b": 2, "c": 3}' >>> json.dumps(testdict, sort_keys=True) '{"a": 1, "b": 2, "c": 3}'
- 73.
- 74. CPU instructions perrequest max Requests Per Second -12% 0% Memory configuration difference? 12%
- 75. if uwsgi.opt.get('optimize_mem', None)== 'True': optimize_mem() b
- 76.
- 77.
- 78. Saving of 30% (oncelery) INSTAGRAM ON PYTHON3 Saving of 12% (on uwsgi/django) CPU MEMORY
- 79. June 2016 Python3 migration Sept2015 Dec 2016 Apr 2017 400M 500M 600M
- 80. Video View Notification SaveDraft Comment Filtering Story Viewer Ranking First Story Notification Self-harm Prevention Live
- 81. MOTIVATION: TYPE HINTS Typehints - 2% done def compose_from_max_id(max_id: Optional[str]) -> Optional[str]: MyPy and typeshed contribution Tooling - collect data and suggest type hints
- 82. MOTIVATION: ASYNC IO Asynchronizeweb framework Parallel network access within a request
- 83. MOTIVATION: COMMUNITY Benchmark webworkload Run time, memory profiling, etc