Your SlideShare is downloading. ×
Python Async IO Horizon
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Python Async IO Horizon

1,308
views

Published on

~10min dive to Python Asynchronous IO …

~10min dive to Python Asynchronous IO

HTML version (recommended): https://dl.dropboxusercontent.com/u/1565687/speak/Python3%20AsyncIO%20Horizon/index.html

Published in: Technology

0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,308
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
21
Comments
0
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 19 Dec 2013 @lukmdo Python Asynchronous I/O asynchronous landscape refresher Photo by Jake Brown used under CC BY 2.0/“Mono+Balance”
  • 2. Agenda ⏰10 minutes ! • Background (can skip) • Problem statement • “Current state” • Python 3 async horizon (by examples)
  • 3. Warmup ! • CPU IO (CPU bound) • Pure IO (IO bound) • C10k • Concurrency vs. parallelism
  • 4. Warmup ! • Lazy schedule a task for later • Avoid waiting for IO (i.e. 2x DB queries) • Solve embarrassingly parallel problems • Control and manage status of task
  • 5. PEP index brief PEP Name Status Doc PEP 255 Simple Generators 2.2 ✔ »» PEP 342 Coroutines via Enhanced Generators 2.5 ✔ » PEP 380 Syntax for Delegating to a Subgenerator 3.3 ✔ » PEP 3148 Futures - execute computations asynchronously 3.2 ✔ » PEP 3153 Asynchronous IO support 3.3 ✔ » PEP 3156 Asynchronous IO Support Rebooted: the "asyncio" Module 3.4 ✔ »
  • 6. PEP index brief PEP Name PEP 219 Stackless Python ❙❙ PEP 220 Coroutines, Generators, Continuations ✗ PEP 319 Python Synchronize/Asynchronize Block ✗ PEP 3145 PEP 3152 Asynchronous I/O For subprocess.Popen Cofunctions Status ❙❙ ❙❙
  • 7. stackless In the Class concurrent.futures threading tornado greenlet subpocess eventlet celery twisted pyuv gevent shed asyncio asyncore
  • 8. In the Class asyncio
  • 9. stackless In the Class concurrent.futures threading tornado greenlet subpocess eventlet celery twisted pyuv gevent shed asyncio asyncore
  • 10. In the Class concurrent.futures threading tornado subpocess celery twisted asyncio
  • 11. Slang* • future ~ result wrapper • coro != generator function (but close) • task = coro wrapped in Future • result = yield from future/coro (to wait for result)
  • 12. new* old style concurrency def update_foo(data): data1 = run_query1(data) data2 = run_query2(data) ! data3 = process(data1, data2) run_update_query(data3) run_update_cache(data3) ! return ["OK"]
  • 13. new* old style concurrency def update_foo(data): data1 = run_query1(data) data2 = run_query2(data) ! data3 = process(data1, data2) run_update_query(data3) run_update_cache(data3) ! return ["OK"]
  • 14. new* old style concurrency
  • 15. new* old style concurrency
  • 16. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): data1 = yield from run_query1(data) data2 = yield from run_query2(data) ! data3 = yield from process(data1, data2) yield from run_update_query(data3) loop.call_soon(update_cache, data3) ! return ["OK"]
  • 17. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): data1 = yield from run_query1(data) data2 = yield from run_query2(data) ! data3 = yield from process(data1, data2) yield from run_update_query(data3) loop.call_soon(update_cache, data3) ! return ["OK"]
  • 18. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): data1 = yield from run_query1(data) data2 = yield from run_query2(data) ! data3 = yield from process(data1, data2) yield from run_update_query(data3) loop.call_soon(update_cache, data3) ! return ["OK"]
  • 19. new* old style concurrency
  • 20. new* old style concurrency
  • 21. new* old style concurrency
  • 22. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): future1 = asyncio.async(run_query1(data)) future2 = asyncio.async(run_query2(data)) ! results = yield from asyncio.gather( future1, future2) data3 = yield from process(*results) yield from run_update_query(data3) ! loop.call_soon(update_cache, data3) ! return ["OK"]
  • 23. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): future1 = asyncio.async(run_query1(data)) future2 = asyncio.async(run_query2(data)) ! results = yield from asyncio.gather( future1, future2) data3 = yield from process(*results) yield from run_update_query(data3) ! loop.call_soon(update_cache, data3) ! return ["OK"]
  • 24. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): future1 = asyncio.async(run_query1(data)) future2 = asyncio.async(run_query2(data)) ! results = yield from asyncio.gather( future1, future2) data3 = yield from process(*results) yield from run_update_query(data3) ! loop.call_soon(update_cache, data3) ! return ["OK"]
  • 25. new* old style concurrency
  • 26. new* old style concurrency
  • 27. new* old style concurrency
  • 28. new* old style concurrency def api_calls(): data1 = call_api1() data2 = call_api2() data3 = call_api3() ! return [data1, data2, data3]
  • 29. new* old style concurrency def api_calls(): data1 = call_api1() data3 = call_api2() data3 = call_api3() ! return [data1, data2, data3]
  • 30. new* old style concurrency
  • 31. new* old style concurrency
  • 32. new* old style concurrency @asyncio.coroutine def api_calls(): future1 = asyncio.Task(call_api1()) future2 = asyncio.Task(call_api2()) future3 = asyncio.Task(call_api3()) ! return ( yield from asyncio.gather( future1, future2, future3) )
  • 33. new* old style concurrency @asyncio.coroutine def api_calls(): future1 = asyncio.Task(call_api1()) future2 = asyncio.Task(call_api2()) future3 = asyncio.Task(call_api3()) ! return ( yield from asyncio.gather( future1, future2, future3) )
  • 34. new* old style concurrency
  • 35. new* old style concurrency
  • 36. new* old style concurrency
  • 37. new* old style concurrency from aiohttp import request ! @asyncio.coroutine def call_api(): url = "URL" response = yield from request('GET', url) return (yield from response.read())
  • 38. Takeaways • concurrency patterns • asyncio bootstrap • early adopters
  • 39. Links • Video by BDFL Tulip: Async I/O for Python 3 • Video by BDFL PyCon US 2013 Keynote • Code by Nikolay Kim aiohttp • Code by google appengine-pipeline • Code by google ndb tasklets Photo by Jake Brown used under CC BY 2.0/“Mono+Tile”

×