Python Performance: Single-threaded, multi-threaded, and Gevent

7,480 views

Published on

A smatter of performance tips for Python programs.

Published in: Technology
0 Comments
13 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,480
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
66
Comments
0
Likes
13
Embeds 0
No embeds

No notes for slide

Python Performance: Single-threaded, multi-threaded, and Gevent

  1. 1. What I Know About Python Performance A. Jesse Jiryu Davis
 http://emptysqua.re
 MongoDB
  2. 2. Agenda • Single-threaded • Multi-threaded • Gevent
  3. 3. Single-Threaded Why is Python slow?
 
 • Everything’s an object
 • Lookup
  4. 4. Everything’s an object def f():! i = 2000! i += 1!
  5. 5. Everything’s an object import dis! dis.dis(f)!
  6. 6. Everything’s an object def f():! i = 2000! i += 1 0 3 6 9 12 13 LOAD_CONST STORE_FAST LOAD_FAST LOAD_CONST INPLACE_ADD! STORE_FAST 1 0 0 2 (2000)! (i)! (i)! (1)! 0 (i)!
  7. 7. Everything’s an object def f():! i = 2000! i += 1 malloc! 0 3 6 9 12 13 LOAD_CONST STORE_FAST LOAD_FAST LOAD_CONST INPLACE_ADD! STORE_FAST 1 0 0 2 (2000)! (i)! malloc! (i)! (1)! 0 (i)!
  8. 8. Everything’s an object 12 seconds for 100M numbers: total = 0! for i in longs:! total += i! print float(total) / len(longs)
  9. 9. Everything’s an object 1 second for 100M numbers: print float(sum(longs)) / len(longs)
  10. 10. Everything’s an object 0.2 seconds for 100M numbers: print numpy.mean(array)
  11. 11. Lookup n = 1! ! def f():! print n!
  12. 12. n = 1! ! def f():! print n! 0 LOAD_GLOBAL 3 PRINT_ITEM! 4 PRINT_NEWLINE hashtable lookup 0 (n)!
  13. 13. n = 1! ! def f(n=n):! print n! 0 LOAD_FAST 3 PRINT_ITEM! 4 PRINT_NEWLINE array offset 0 (n)!
  14. 14. Profiling
  15. 15. Profiling import yappi! ! yappi.start(builtins=True)! my_function()! yappi.stop()! stats = yappi.get_func_stats()! stats.save('yappi.callgrind', type='callgrind')!
  16. 16. KCacheGrind
  17. 17. KCacheGrind
  18. 18. Multi-threaded
  19. 19. Multi-threaded When is the GIL released?
 
 • Every 100 bytecodes
 • Acquiring other locks
 • Network or file operations
  20. 20. Multi-threaded Because: Py_BEGIN_ALLOW_THREADS! outlen = recv(s->sock_fd, cbuf, len);! Py_END_ALLOW_THREADS! socketmodule.c
  21. 21. Multi-threaded Because: Py_BEGIN_ALLOW_THREADS! outlen = recv(s->sock_fd, cbuf, len);! Py_END_ALLOW_THREADS! release the GIL! socketmodule.c
  22. 22. Multi-threaded This makes sense: threads = [! threading.Thread(target=download_a_url())! for _ in range(10)]! ! for t in threads:! t.start()! ! for t in threads:! t.join()!
  23. 23. Multi-threaded This makes no sense: threads = [! threading.Thread(target=do_calculations())! for _ in range(10)]! ! for t in threads:! t.start()! ! for t in threads:! t.join()!
  24. 24. Gevent from gevent import monkey! monkey.patch_all()! ! threads = [! threading.Thread(target=download_a_url())]! ! for t in threads:! t.start()! ! for t in threads:! t.join()! actually a greenlet
  25. 25. GreenletProfiler
  26. 26. Links http://bit.ly/pythonperformance

×