Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Python Yield

4,565 views

Published on

Python yield

Published in: Technology
  • Be the first to comment

Python Yield

  1. 1. Python yield Mail & Popo: yangjunwei@ 1
  2. 2. Agenda• why thinking in yield?• yield• yield• yield 2
  3. 3. why thinking in yield? 3
  4. 4. why thinking in yield?db.begin()if check_error1_happen(db): db.rollback() return Response(“error1 happens”)if check_error2_happen(db): db.rollback() return Response(“error2 happens”)db.table.update()....db.commit()return Response(“success”) 4
  5. 5. why thinking in yield?• pythonic• bug 5
  6. 6. why thinking in yield? @contextmanager def transaction(conn):class Job(object): trans = Job() def __init__(self): conn.begin() self._finished = False try: yield trans def is_finished(self): except: return self._finished conn.rollback() raise def finish(self): if trans.is_finished(): self._finished = True conn.commit() else: conn.rollback() 6
  7. 7. why thinking in yield?with transaction(db) as trans: if check_error1_happen(db): return Response(“error1 happens”) if check_error2_happen(db): return Response(“error2 happens”) db.table.update() .... trans.finish()return Response(“success”) 7
  8. 8. why thinking in yield?main main A yield function B C yield function 8
  9. 9. yield 9
  10. 10. yieldUsing a yield expression in a function definition is sufficient tocause that definition to create a generator function instead ofa normal function. 10
  11. 11. yieldUsing a yield expression in a function definition is sufficient tocause that definition to create a generator function instead ofa normal function. >>> def f(): ... yield "hello world" ... >>> type(f()) <type generator> 10
  12. 12. yieldThat generator then controls the execution of a generatorfunction. 11
  13. 13. yieldThat generator then controls the execution of a generatorfunction. • send() • next() ==> send(None) • throw() • close() ==> throw(GeneratorExit) 11
  14. 14. yieldyield is a expression, not a statement. New in python 2.5 12
  15. 15. yieldyield is a expression, not a statement. New in python 2.5>>> def f():... a = yield "hello"... print a...>>> b = f()>>> b.send(None)hello>>> b.send("world")worldStopIteration 12
  16. 16. yieldgenerator is also a iterator. 13
  17. 17. yieldgenerator is also a iterator.>>> a = f()>>> a is a.__iter__()True 13
  18. 18. yieldgenerator object 14
  19. 19. yieldgenerator object>>> def f():... yield "hello world"...>>> a = f()>>> a.next()“hello world”>>> a.next()StopIteration>>> a.next()StopIteration 14
  20. 20. yield 15
  21. 21. yieldtstate_head next next PyThreadState PyThreadState ... frame PyFrameObject f_back PyFrameObject f_back python ... 16
  22. 22. yield code block compile PyCodeObject MAKE_FUNCTIONPyFunctionObject CALL_FUNCTION PyFrameObject 17
  23. 23. yield[frameobject.c]typedef struct { ... struct _frame *f_back; PyCodeObject *f_code; PyObject *f_builtins; PyObject *f_globals; PyObject *f_locals; PyObject *f_valuestack; PyObject *f_stacktop; ... int f_lasti; ...} PyFrameObject; 18
  24. 24. yield[codeobject.c]typedef struct { ... int co_flags; ... PyObject *co_code; PyObject *co_consts; PyObject *co_names; ...} PyCodeObject; 19
  25. 25. yield[a.py] [b.py]def f(): def f(): return “hello world” yield “hello world”f() f() PyCodeObject <==> code block 20
  26. 26. yield4 0 LOAD_CONST 0 (<code object f ..>) 3 MAKE_FUNCTION 0 6 STORE_NAME 0 (f)7 9 LOAD_NAME 0 (f) 12 CALL_FUNCTION 0 15 POP_TOP 16 LOAD_CONST 1 (None) 19 RETURN_VALUE dis.dis(main_code) 21
  27. 27. yield5 0 LOAD_CONST 1 (hello world) 3 RETURN_VALUE5 0 LOAD_CONST 1 (hello world) 3 YIELD_VALUE 4 POP_TOP 5 LOAD_CONST 0 (None) 8 RETURN_VALUE dis.dis(f_code) 22
  28. 28. yield return generator 23
  29. 29. yield[ceval.c] CALL_FUNCTIONf = PyFrame_New(tstate, co, globals, locals);...if (co->co_flags & CO_GENERATOR) { ... f->f_back = NULL; ... return PyGen_New(f);}retval = PyEval_EvalFrameEx(f,0); 24
  30. 30. yield[frameobject.c]PyFrame_New(PyThreadState *tstate,PyCodeObject *code, PyObject *globals, PyObject *locals){ ... PyFrameObject *back = tstate->frame; PyFrameObject *f; f = PyObject_GC_Resize(PyFrameObject, f, extras); ... f->f_code = code; ... f->f_back = back; f->f_tstate = tstate; return f;} 25
  31. 31. yield[ceval.c]PyObject *PyEval_EvalFrameEx(PyFrameObject *f, int throwflag){ ... PyThreadState *tstate = PyThreadState_GET(); tstate->frame = f; ... // exec opcode ... tstate->frame = f->f_back; return retval;} 26
  32. 32. yieldtstate_head next PyThreadState ... frame A frame f_back B frame f_back ... 27
  33. 33. yieldtstate_head next PyThreadState ... frame A frame new frame f_back B frame f_back ... 27
  34. 34. yieldtstate_head next PyThreadState ... frame A frame new frame f_back B frame f_back ... 27
  35. 35. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... 27
  36. 36. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... 27
  37. 37. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... 27
  38. 38. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... 27
  39. 39. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... 27
  40. 40. yieldtstate_head next PyThreadState ... frame A frame f_back B frame f_back ... yield 28
  41. 41. yieldtstate_head next PyThreadState ... frame A frame new frame f_back B frame f_back ... yield 28
  42. 42. yieldtstate_head next PyThreadState ... frame A frame new frame f_back B frame f_back ... yield 28
  43. 43. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... yield 28
  44. 44. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... yield 28
  45. 45. yieldtstate_head next PyThreadState ... frame A frame new frame f_back B frame f_back ... yield 28
  46. 46. yield[genobject.c]static PyObject * gen_send_ex(PyGenObject *gen, PyObject *arg, intexc) { PyFrameObject *f = gen->gi_frame; f->f_back = tstate->frame; ... if (f->f_lasti == -1) { .... } else { result = arg ? arg : Py_None; Py_INCREF(result); *(f->f_stacktop++) = result; } ... result = PyEval_EvalFrameEx(f, exc);} 29
  47. 47. yieldtstate_head next PyThreadState ... frame A frame(caller) new frame f_back B frame(creator) f_back ... send 30
  48. 48. yieldtstate_head next PyThreadState ... frame A frame(caller) new frame f_back B frame(creator) f_back ... send 30
  49. 49. yieldtstate_head next PyThreadState ... frame A frame(caller) new frame f_back f_back B frame(creator) f_back ... send 30
  50. 50. yieldtstate_head next PyThreadState ... frame A frame(caller) new frame f_back f_back B frame(creator) f_back ... send 30
  51. 51. yieldtstate_head next PyThreadState ... frame A frame(caller) new frame f_back f_back B frame(creator) f_back ... send 30
  52. 52. yieldresume frame 31
  53. 53. yield[ceval.c]PyObject *PyEval_EvalFrameEx(PyFrameObject *f, int throwflag){ ... next_instr = first_instr + f->f_lasti + 1; stack_pointer = f->f_stacktop; assert(stack_pointer != NULL); f->f_stacktop = NULL; ... case YIELD_VALUE: retval = POP(); f->f_stacktop = stack_pointer; why = WHY_YIELD; goto fast_yield;} 32
  54. 54. yield[genobject.c]static PyObject *gen_throw(PyGenObject *gen,PyObject *args){ ... PyErr_Restore(typ, val, tb); return gen_send_ex(gen, Py_None, 1);} throw 33
  55. 55. yielddef close(self): try: self.throw(GeneratorExit) except (GeneratorExit, StopIteration): pass else: raise RuntimeError("generator ignored GeneratorExit") # Other exceptions are not caught close 34
  56. 56. yield 35
  57. 57. • xrange• with_statement“ ”• python Trampoline• generator coroutine 36
  58. 58. xrange 37
  59. 59. xrange[rangeobject.c]static PyObject *rangeiter_next(rangeiterobject *r){ if (r->index < r->len) return PyInt_FromLong(r->start + (r->index++) * r->step); return NULL;} xrange 38
  60. 60. xrangedef xrange(start, stop, step=1): i = start - step while True: i = i + step if i < stop: yield i else: return yield 39
  61. 61. xrangel = [x for x in xrange(10)]g = (x * x for x in xrange(10)) list generator 40
  62. 62. xrangel = [x for x in xrange(10)]g = (x * x for x in xrange(10)) list generator 40
  63. 63. xrangel = [x for x in xrange(10)]g = (x * x for x in xrange(10)) list generatordef f(): for x in xrange(10): yield x * x 40
  64. 64. with_statement“ ” 41
  65. 65. “with_statement” with with 42
  66. 66. “with_statement” with withwith EXPR as VAR: BLOCK 42
  67. 67. “with_statement” with withwith EXPR as VAR: BLOCK 42
  68. 68. “with_statement”mgr = (EXPR)exit = type(mgr).__exit__ # Not calling it yetvalue = type(mgr).__enter__(mgr)exc = Truetry: try: VAR = value # Only if "as VAR" is present BLOCK except: exc = False if not exit(mgr, *sys.exc_info()): raisefinally: if exc: exit(mgr, None, None, None) 43
  69. 69. “with_statement”class Transaction(object): def __init__(self, db): self._db = db self._trans = Job() def __enter__(self): self._db.begin() return self._trans def __exit__(self, type, value, traceback): if type is not None: self._db.rollback() else: self._db.commit() yield transaction 44
  70. 70. “with_statement”@contextmanagerdef transaction(conn): trans = Job() conn.begin() try: yield trans except: conn.rollback() raise if trans.is_finished(): conn.commit() else: conn.rollback() 45
  71. 71. “with_statement”@contextmanagerdef transaction(conn): trans = Job() conn.begin() try: yield trans except: conn.rollback() raise if trans.is_finished(): conn.commit() else: conn.rollback() 45
  72. 72. python Trampoline 46
  73. 73. python Trampolinedef f(n): if n < 2: return n else: return n + f(n - 1)>>> f(999)499500>>> f(1000)RuntimeError: maximum recursion depth exceeded 47
  74. 74. python Trampolinesys.getrecursionlimit()Return the current value of the recursion limit, themaximum depth of the Python interpreter stack. Thislimit prevents infinite recursion from causing anoverflow of the C stack and crashing Python. It can beset by setrecursionlimit(). 48
  75. 75. python Trampoline 49
  76. 76. python Trampoline f(n) f(n) control f(n-1) f(n-1) f(n-2) scheduler f(n-2)f_back ... ... f(1) f(1) 50
  77. 77. python Trampoline• let generator "call" other generator by yielding the generator they wish to invoke.• Any non-generator value yielded by a generator is returned to the generator that "called" the one yielding the value. 51
  78. 78. python Trampolinedef f(n): if n < 2: yield n else: yield n + (yield f(n-1)) 52
  79. 79. python Trampolinedef trampoline(main): stack = [] value = main while True: if type(value) is types.GeneratorType: stack.append(value) value = stack[-1].next() else: stack.pop() if stack: value = stack[-1].send(value) else: return value 53
  80. 80. generator coroutine 54
  81. 81. generator coroutineCoroutine( )" " " " 55
  82. 82. generator coroutineCoroutine( )" " " " Pythons generator functions are almost coroutines. 55
  83. 83. generator coroutinegenerator coroutine 56
  84. 84. generator coroutinegenerator coroutine• coroutine “call” coroutine• coroutine scheduler 56
  85. 85. generator coroutinecoroutine-based Python networking library• Eventlet(http://eventlet.net/)• Gevent(http://www.gevent.org/) 57
  86. 86. Resources & References• python references: yield expression• python• PEP 342: Coroutines via Enhanced Generators• PEP 343: the with statement 58
  87. 87. Q&A 59
  88. 88. Thanks! 60

×