Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Allison Kaptur: Bytes in the Machine: Inside the CPython interpreter, PyGotham 2014

1,004 views

Published on

Byterun is a Python interpreter written in Python with Ned Batchelder. It's architected to mirror the structure of CPython (and be more readable, too)! Learn how the interpreter is constructed, how ignorant the Python compiler is, and how you use a 1,500 line switch statement every day.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Allison Kaptur: Bytes in the Machine: Inside the CPython interpreter, PyGotham 2014

  1. 1. Byterun: A (C)Python interpreter in Python Allison Kaptur ! github.com/akaptur akaptur.github.io @akaptur
  2. 2. Byterun with Ned Batchelder ! Based on # pyvm2 by Paul Swartz (z3p) from http://www.twistedmatrix.com/users/z3p/
  3. 3. Why would you do such a thing >>> if a or b: ... do_stuff()
  4. 4. Some things we can do out = "" for i in range(5): out = out + str(i) print(out)
  5. 5. Some things we can do def fn(a, b=17, c="Hello", d=[]): d.append(99) print(a, b, c, d) ! fn(1) fn(2, 3) fn(3, c="Bye") fn(4, d=["What?"]) fn(5, "b", "c")
  6. 6. Some things we can do def verbose(func): def _wrapper(*args, **kwargs): return func(*args, **kwargs) return _wrapper ! @verbose def add(x, y): return x+y ! add(7, 3)
  7. 7. Some things we can do try: raise ValueError("oops") except ValueError as e: print("Caught: %s" % e) print("All done")
  8. 8. Some things we can do class NullContext(object): def __enter__(self): l.append('i') return self ! def __exit__(self, exc_type, exc_val, exc_tb): l.append('o') return False ! l = [] for i in range(3): with NullContext(): l.append('w') if i % 2: break l.append('z') l.append('e') ! l.append('r') s = ''.join(l) print("Look: %r" % s) assert s == "iwzoeiwor"
  9. 9. Some things we can do g = (x*x for x in range(3)) print(list(g))
  10. 10. A problem g = (x*x for x in range(5)) h = (y+1 for y in g) print(list(h))
  11. 11. The Python virtual machine: ! A bytecode interpreter
  12. 12. Bytecode: the internal representation of a python program in the interpreter
  13. 13. Bytecode: it’s bytes! >>> def mod(a, b): ... ans = a % b ... return ans
  14. 14. Bytecode: it’s bytes! >>> def mod(a, b): ... ans = a % b ... return ans >>> mod.func_code.co_code Function Code object Bytecode
  15. 15. Bytecode: it’s bytes! >>> def mod(a, b): ... ans = a % b ... return ans >>> mod.func_code.co_code '|x00x00|x01x00x16}x02x00|x02x00S'
  16. 16. Bytecode: it’s bytes! >>> def mod(a, b): ... ans = a % b ... return ans >>> mod.func_code.co_code ‘|x00x00|x01x00x16}x02x00|x02x00S' >>> [ord(b) for b in mod.func_code.co_code] [124, 0, 0, 124, 1, 0, 22, 125, 2, 0, 124, 2, 0, 83]
  17. 17. dis, a bytecode disassembler >>> import dis >>> dis.dis(mod) 2 0 LOAD_FAST 0 (a) 3 LOAD_FAST 1 (b) 6 BINARY_MODULO 7 STORE_FAST 2 (ans) ! 3 10 LOAD_FAST 2 (ans) 13 RETURN_VALUE
  18. 18. dis, a bytecode disassembler >>> import dis >>> dis.dis(mod) 2 0 LOAD_FAST 0 (a) 3 LOAD_FAST 1 (b) 6 BINARY_MODULO 7 STORE_FAST 2 (ans) ! 3 10 LOAD_FAST 2 (ans) 13 RETURN_VALUE Line Number Index in bytecode Instruction name, for humans More bytes, the argument to each instruction Hint about arguments
  19. 19. whatever some other thing something whatever some other thing something a b whatever some other thing something ans Before After BINARY_MODULO After LOAD_FAST Data stack on a frame
  20. 20. def foo(): x = 1 def bar(y): z = y + 2 return z return bar(x) foo() # <--- (1) ! c a l l ! s t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [<foo>] k ---------------------
  21. 21. def foo(): x = 1 def bar(y): z = y + 2 return z return bar(x) # <--- (2) foo() # <--- (1) ! c a l l --------------------- | foo Frame | -> blocks: [] s | | -> data: [<bar>, 1] t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [] k ---------------------
  22. 22. def foo(): x = 1 def bar(y): z = y + 2 # <--- (3) return z return bar(x) # <--- (2) foo() # <--- (1) ! c --------------------- a | bar Frame | -> blocks: [] l | (newest) | -> data: [1, 2] l --------------------- | foo Frame | -> blocks: [] s | | -> data: [] t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [] k ---------------------
  23. 23. def foo(): x = 1 def bar(y): z = y + 2 # <--- (3) return z return bar(x) # <--- (2) foo() # <--- (1) ! c --------------------- a | bar Frame | -> blocks: [] l | (newest) | -> data: [3] l --------------------- | foo Frame | -> blocks: [] s | | -> data: [] t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [] k ---------------------
  24. 24. def foo(): x = 1 def bar(y): z = y + 2 # <--- (3) return z return bar(x) # <--- (2) foo() # <--- (1) ! c a l l --------------------- | foo Frame | -> blocks: [] s | | -> data: [3] t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [] k ---------------------
  25. 25. def foo(): x = 1 def bar(y): z = y + 2 # <--- (3) return z return bar(x) # <--- (2) foo() # <--- (1) ! c a l l ! s t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [3] k ---------------------
  26. 26. dis, a bytecode disassembler >>> import dis >>> dis.dis(mod) 2 0 LOAD_FAST 0 (a) 3 LOAD_FAST 1 (b) 6 BINARY_MODULO 7 STORE_FAST 2 (ans) ! 3 10 LOAD_FAST 2 (ans) 13 RETURN_VALUE
  27. 27. } /*switch*/ /* Main switch on opcode */ READ_TIMESTAMP(inst0); ! switch (opcode) {
  28. 28. #ifdef CASE_TOO_BIG default: switch (opcode) { #endif /* Turn this on if your compiler chokes on the big switch: */ /* #define CASE_TOO_BIG 1 */
  29. 29. Back to that bytecode ! >>> dis.dis(mod) 2 0 LOAD_FAST 0 (a) 3 LOAD_FAST 1 (b) 6 BINARY_MODULO 7 STORE_FAST 2 (ans) ! 3 10 LOAD_FAST 2 (ans) 13 RETURN_VALUE
  30. 30. case LOAD_FAST: x = GETLOCAL(oparg); if (x != NULL) { Py_INCREF(x); PUSH(x); goto fast_next_opcode; } format_exc_check_arg(PyExc_UnboundLocalError, UNBOUNDLOCAL_ERROR_MSG, PyTuple_GetItem(co->co_varnames, oparg)); break;
  31. 31. case BINARY_MODULO: w = POP(); v = TOP(); if (PyString_CheckExact(v)) x = PyString_Format(v, w); else x = PyNumber_Remainder(v, w); Py_DECREF(v); Py_DECREF(w); SET_TOP(x); if (x != NULL) continue; break;
  32. 32. It’s “dynamic” >>> def mod(a, b): ... ans = a % b ... return ans >>> mod(15, 4) 3
  33. 33. “Dynamic” >>> def mod(a, b): ... ans = a % b ... return ans >>> mod(15, 4) 3 >>> mod(“%s%s”, (“Py”, “Gotham”))
  34. 34. “Dynamic” >>> def mod(a, b): ... ans = a % b ... return ans >>> mod(15, 4) 3 >>> mod(“%s%s”, (“Py”, “Gotham”)) PyGotham
  35. 35. “Dynamic” >>> def mod(a, b): ... ans = a % b ... return ans >>> mod(15, 4) 3 >>> mod(“%s%s”, (“Py”, “Gotham”)) PyGotham >>> print “%s%s” % (“Py”, “Gotham”) PyGotham
  36. 36. case BINARY_MODULO: w = POP(); v = TOP(); if (PyString_CheckExact(v)) x = PyString_Format(v, w); else x = PyNumber_Remainder(v, w); Py_DECREF(v); Py_DECREF(w); SET_TOP(x); if (x != NULL) continue; break;
  37. 37. >>> class Surprising(object): … def __mod__(self, other): … print “Surprise!” ! >>> s = Surprising() >>> t = Surprsing() >>> s % t Surprise!
  38. 38. “In the general absence of type information, almost every instruction must be treated as INVOKE_ARBITRARY_METHOD.” ! - Russell Power and Alex Rubinsteyn, “How Fast Can We Make Interpreted Python?”
  39. 39. Back to our problem g = (x*x for x in range(5)) h = (y+1 for y in g) print(list(h))
  40. 40. def foo(): x = 1 def bar(y): z = y + 2 return z return bar(x) foo() # <--- (1) ! c a l l ! s t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [<foo>] k ---------------------
  41. 41. def foo(): x = 1 def bar(y): z = y + 2 return z return bar(x) # <--- (2) foo() # <--- (1) ! c a l l --------------------- | foo Frame | -> blocks: [] s | | -> data: [<bar>, 1] t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [] k ---------------------
  42. 42. def foo(): x = 1 def bar(y): z = y + 2 # <--- (3) return z return bar(x) # <--- (2) foo() # <--- (1) ! c --------------------- a | bar Frame | -> blocks: [] l | (newest) | -> data: [1, 2] l --------------------- | foo Frame | -> blocks: [] s | | -> data: [] t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [] k ---------------------
  43. 43. def foo(): x = 1 def bar(y): z = y + 2 # <--- (3) return z return bar(x) # <--- (2) foo() # <--- (1) ! c --------------------- a | bar Frame | -> blocks: [] l | (newest) | -> data: [3] l --------------------- | foo Frame | -> blocks: [] s | | -> data: [] t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [] k ---------------------
  44. 44. def foo(): x = 1 def bar(y): z = y + 2 # <--- (3) return z return bar(x) # <--- (2) foo() # <--- (1) ! c a l l --------------------- | foo Frame | -> blocks: [] s | | -> data: [3] t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [] k ---------------------
  45. 45. def foo(): x = 1 def bar(y): z = y + 2 # <--- (3) return z return bar(x) # <--- (2) foo() # <--- (1) ! c a l l ! s t --------------------- a | main (module) Frame | -> blocks: [] c | (oldest) | -> data: [3] k ---------------------
  46. 46. Back to our problem g = (x*x for x in range(5)) h = (y+1 for y in g) print(list(h))
  47. 47. More Great blogs http://tech.blog.aknin.name/category/my-projects/ pythons-innards/ by @aknin http://eli.thegreenplace.net/ by Eli Bendersky ! Contribute! Find bugs! https://github.com/nedbat/byterun ! Apply to Hacker School! www.hackerschool.com/apply

×