Python Yield

4,290 views
4,103 views

Published on

Python yield

Published in: Technology
0 Comments
14 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,290
On SlideShare
0
From Embeds
0
Number of Embeds
478
Actions
Shares
0
Downloads
146
Comments
0
Likes
14
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Python Yield

    1. 1. Python yield Mail & Popo: yangjunwei@ 1
    2. 2. Agenda• why thinking in yield?• yield• yield• yield 2
    3. 3. why thinking in yield? 3
    4. 4. why thinking in yield?db.begin()if check_error1_happen(db): db.rollback() return Response(“error1 happens”)if check_error2_happen(db): db.rollback() return Response(“error2 happens”)db.table.update()....db.commit()return Response(“success”) 4
    5. 5. why thinking in yield?• pythonic• bug 5
    6. 6. why thinking in yield? @contextmanager def transaction(conn):class Job(object): trans = Job() def __init__(self): conn.begin() self._finished = False try: yield trans def is_finished(self): except: return self._finished conn.rollback() raise def finish(self): if trans.is_finished(): self._finished = True conn.commit() else: conn.rollback() 6
    7. 7. why thinking in yield?with transaction(db) as trans: if check_error1_happen(db): return Response(“error1 happens”) if check_error2_happen(db): return Response(“error2 happens”) db.table.update() .... trans.finish()return Response(“success”) 7
    8. 8. why thinking in yield?main main A yield function B C yield function 8
    9. 9. yield 9
    10. 10. yieldUsing a yield expression in a function definition is sufficient tocause that definition to create a generator function instead ofa normal function. 10
    11. 11. yieldUsing a yield expression in a function definition is sufficient tocause that definition to create a generator function instead ofa normal function. >>> def f(): ... yield "hello world" ... >>> type(f()) <type generator> 10
    12. 12. yieldThat generator then controls the execution of a generatorfunction. 11
    13. 13. yieldThat generator then controls the execution of a generatorfunction. • send() • next() ==> send(None) • throw() • close() ==> throw(GeneratorExit) 11
    14. 14. yieldyield is a expression, not a statement. New in python 2.5 12
    15. 15. yieldyield is a expression, not a statement. New in python 2.5>>> def f():... a = yield "hello"... print a...>>> b = f()>>> b.send(None)hello>>> b.send("world")worldStopIteration 12
    16. 16. yieldgenerator is also a iterator. 13
    17. 17. yieldgenerator is also a iterator.>>> a = f()>>> a is a.__iter__()True 13
    18. 18. yieldgenerator object 14
    19. 19. yieldgenerator object>>> def f():... yield "hello world"...>>> a = f()>>> a.next()“hello world”>>> a.next()StopIteration>>> a.next()StopIteration 14
    20. 20. yield 15
    21. 21. yieldtstate_head next next PyThreadState PyThreadState ... frame PyFrameObject f_back PyFrameObject f_back python ... 16
    22. 22. yield code block compile PyCodeObject MAKE_FUNCTIONPyFunctionObject CALL_FUNCTION PyFrameObject 17
    23. 23. yield[frameobject.c]typedef struct { ... struct _frame *f_back; PyCodeObject *f_code; PyObject *f_builtins; PyObject *f_globals; PyObject *f_locals; PyObject *f_valuestack; PyObject *f_stacktop; ... int f_lasti; ...} PyFrameObject; 18
    24. 24. yield[codeobject.c]typedef struct { ... int co_flags; ... PyObject *co_code; PyObject *co_consts; PyObject *co_names; ...} PyCodeObject; 19
    25. 25. yield[a.py] [b.py]def f(): def f(): return “hello world” yield “hello world”f() f() PyCodeObject <==> code block 20
    26. 26. yield4 0 LOAD_CONST 0 (<code object f ..>) 3 MAKE_FUNCTION 0 6 STORE_NAME 0 (f)7 9 LOAD_NAME 0 (f) 12 CALL_FUNCTION 0 15 POP_TOP 16 LOAD_CONST 1 (None) 19 RETURN_VALUE dis.dis(main_code) 21
    27. 27. yield5 0 LOAD_CONST 1 (hello world) 3 RETURN_VALUE5 0 LOAD_CONST 1 (hello world) 3 YIELD_VALUE 4 POP_TOP 5 LOAD_CONST 0 (None) 8 RETURN_VALUE dis.dis(f_code) 22
    28. 28. yield return generator 23
    29. 29. yield[ceval.c] CALL_FUNCTIONf = PyFrame_New(tstate, co, globals, locals);...if (co->co_flags & CO_GENERATOR) { ... f->f_back = NULL; ... return PyGen_New(f);}retval = PyEval_EvalFrameEx(f,0); 24
    30. 30. yield[frameobject.c]PyFrame_New(PyThreadState *tstate,PyCodeObject *code, PyObject *globals, PyObject *locals){ ... PyFrameObject *back = tstate->frame; PyFrameObject *f; f = PyObject_GC_Resize(PyFrameObject, f, extras); ... f->f_code = code; ... f->f_back = back; f->f_tstate = tstate; return f;} 25
    31. 31. yield[ceval.c]PyObject *PyEval_EvalFrameEx(PyFrameObject *f, int throwflag){ ... PyThreadState *tstate = PyThreadState_GET(); tstate->frame = f; ... // exec opcode ... tstate->frame = f->f_back; return retval;} 26
    32. 32. yieldtstate_head next PyThreadState ... frame A frame f_back B frame f_back ... 27
    33. 33. yieldtstate_head next PyThreadState ... frame A frame new frame f_back B frame f_back ... 27
    34. 34. yieldtstate_head next PyThreadState ... frame A frame new frame f_back B frame f_back ... 27
    35. 35. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... 27
    36. 36. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... 27
    37. 37. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... 27
    38. 38. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... 27
    39. 39. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... 27
    40. 40. yieldtstate_head next PyThreadState ... frame A frame f_back B frame f_back ... yield 28
    41. 41. yieldtstate_head next PyThreadState ... frame A frame new frame f_back B frame f_back ... yield 28
    42. 42. yieldtstate_head next PyThreadState ... frame A frame new frame f_back B frame f_back ... yield 28
    43. 43. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... yield 28
    44. 44. yieldtstate_head next PyThreadState ... frame A frame new frame f_back f_back B frame f_back ... yield 28
    45. 45. yieldtstate_head next PyThreadState ... frame A frame new frame f_back B frame f_back ... yield 28
    46. 46. yield[genobject.c]static PyObject * gen_send_ex(PyGenObject *gen, PyObject *arg, intexc) { PyFrameObject *f = gen->gi_frame; f->f_back = tstate->frame; ... if (f->f_lasti == -1) { .... } else { result = arg ? arg : Py_None; Py_INCREF(result); *(f->f_stacktop++) = result; } ... result = PyEval_EvalFrameEx(f, exc);} 29
    47. 47. yieldtstate_head next PyThreadState ... frame A frame(caller) new frame f_back B frame(creator) f_back ... send 30
    48. 48. yieldtstate_head next PyThreadState ... frame A frame(caller) new frame f_back B frame(creator) f_back ... send 30
    49. 49. yieldtstate_head next PyThreadState ... frame A frame(caller) new frame f_back f_back B frame(creator) f_back ... send 30
    50. 50. yieldtstate_head next PyThreadState ... frame A frame(caller) new frame f_back f_back B frame(creator) f_back ... send 30
    51. 51. yieldtstate_head next PyThreadState ... frame A frame(caller) new frame f_back f_back B frame(creator) f_back ... send 30
    52. 52. yieldresume frame 31
    53. 53. yield[ceval.c]PyObject *PyEval_EvalFrameEx(PyFrameObject *f, int throwflag){ ... next_instr = first_instr + f->f_lasti + 1; stack_pointer = f->f_stacktop; assert(stack_pointer != NULL); f->f_stacktop = NULL; ... case YIELD_VALUE: retval = POP(); f->f_stacktop = stack_pointer; why = WHY_YIELD; goto fast_yield;} 32
    54. 54. yield[genobject.c]static PyObject *gen_throw(PyGenObject *gen,PyObject *args){ ... PyErr_Restore(typ, val, tb); return gen_send_ex(gen, Py_None, 1);} throw 33
    55. 55. yielddef close(self): try: self.throw(GeneratorExit) except (GeneratorExit, StopIteration): pass else: raise RuntimeError("generator ignored GeneratorExit") # Other exceptions are not caught close 34
    56. 56. yield 35
    57. 57. • xrange• with_statement“ ”• python Trampoline• generator coroutine 36
    58. 58. xrange 37
    59. 59. xrange[rangeobject.c]static PyObject *rangeiter_next(rangeiterobject *r){ if (r->index < r->len) return PyInt_FromLong(r->start + (r->index++) * r->step); return NULL;} xrange 38
    60. 60. xrangedef xrange(start, stop, step=1): i = start - step while True: i = i + step if i < stop: yield i else: return yield 39
    61. 61. xrangel = [x for x in xrange(10)]g = (x * x for x in xrange(10)) list generator 40
    62. 62. xrangel = [x for x in xrange(10)]g = (x * x for x in xrange(10)) list generator 40
    63. 63. xrangel = [x for x in xrange(10)]g = (x * x for x in xrange(10)) list generatordef f(): for x in xrange(10): yield x * x 40
    64. 64. with_statement“ ” 41
    65. 65. “with_statement” with with 42
    66. 66. “with_statement” with withwith EXPR as VAR: BLOCK 42
    67. 67. “with_statement” with withwith EXPR as VAR: BLOCK 42
    68. 68. “with_statement”mgr = (EXPR)exit = type(mgr).__exit__ # Not calling it yetvalue = type(mgr).__enter__(mgr)exc = Truetry: try: VAR = value # Only if "as VAR" is present BLOCK except: exc = False if not exit(mgr, *sys.exc_info()): raisefinally: if exc: exit(mgr, None, None, None) 43
    69. 69. “with_statement”class Transaction(object): def __init__(self, db): self._db = db self._trans = Job() def __enter__(self): self._db.begin() return self._trans def __exit__(self, type, value, traceback): if type is not None: self._db.rollback() else: self._db.commit() yield transaction 44
    70. 70. “with_statement”@contextmanagerdef transaction(conn): trans = Job() conn.begin() try: yield trans except: conn.rollback() raise if trans.is_finished(): conn.commit() else: conn.rollback() 45
    71. 71. “with_statement”@contextmanagerdef transaction(conn): trans = Job() conn.begin() try: yield trans except: conn.rollback() raise if trans.is_finished(): conn.commit() else: conn.rollback() 45
    72. 72. python Trampoline 46
    73. 73. python Trampolinedef f(n): if n < 2: return n else: return n + f(n - 1)>>> f(999)499500>>> f(1000)RuntimeError: maximum recursion depth exceeded 47
    74. 74. python Trampolinesys.getrecursionlimit()Return the current value of the recursion limit, themaximum depth of the Python interpreter stack. Thislimit prevents infinite recursion from causing anoverflow of the C stack and crashing Python. It can beset by setrecursionlimit(). 48
    75. 75. python Trampoline 49
    76. 76. python Trampoline f(n) f(n) control f(n-1) f(n-1) f(n-2) scheduler f(n-2)f_back ... ... f(1) f(1) 50
    77. 77. python Trampoline• let generator "call" other generator by yielding the generator they wish to invoke.• Any non-generator value yielded by a generator is returned to the generator that "called" the one yielding the value. 51
    78. 78. python Trampolinedef f(n): if n < 2: yield n else: yield n + (yield f(n-1)) 52
    79. 79. python Trampolinedef trampoline(main): stack = [] value = main while True: if type(value) is types.GeneratorType: stack.append(value) value = stack[-1].next() else: stack.pop() if stack: value = stack[-1].send(value) else: return value 53
    80. 80. generator coroutine 54
    81. 81. generator coroutineCoroutine( )" " " " 55
    82. 82. generator coroutineCoroutine( )" " " " Pythons generator functions are almost coroutines. 55
    83. 83. generator coroutinegenerator coroutine 56
    84. 84. generator coroutinegenerator coroutine• coroutine “call” coroutine• coroutine scheduler 56
    85. 85. generator coroutinecoroutine-based Python networking library• Eventlet(http://eventlet.net/)• Gevent(http://www.gevent.org/) 57
    86. 86. Resources & References• python references: yield expression• python• PEP 342: Coroutines via Enhanced Generators• PEP 343: the with statement 58
    87. 87. Q&A 59
    88. 88. Thanks! 60

    ×