19 Dec 2013
@lukmdo

Python Asynchronous I/O
asynchronous landscape refresher

Photo by Jake Brown used under CC BY 2.0/“M...
Agenda ⏰10 minutes
!

•

Background (can skip)

•

Problem statement

•

“Current state”

•

Python 3 async horizon (by ex...
Warmup
!

•

CPU IO (CPU bound)

•

Pure IO (IO bound)

•

C10k

•

Concurrency vs. parallelism
Warmup
!

•

Lazy schedule a task for later

•

Avoid waiting for IO (i.e. 2x DB queries)

•

Solve embarrassingly paralle...
PEP index brief
PEP

Name

Status

Doc

PEP 255

Simple Generators

2.2 ✔

»»

PEP 342

Coroutines via Enhanced Generators...
PEP index brief
PEP

Name

PEP 219

Stackless Python

❙❙

PEP 220

Coroutines, Generators, Continuations

✗

PEP 319

Pyth...
stackless

In the Class
concurrent.futures
threading
tornado
greenlet
subpocess
eventlet
celery
twisted
pyuv
gevent
shed a...
In the Class

asyncio
stackless

In the Class
concurrent.futures
threading
tornado
greenlet
subpocess
eventlet
celery
twisted
pyuv
gevent
shed a...
In the Class
concurrent.futures
threading
tornado
subpocess
celery
twisted
asyncio
Slang*
•

future ~ result wrapper

•

coro != generator function (but close)

•

task = coro wrapped in Future

•

result ...
new* old style concurrency
def update_foo(data):
data1 = run_query1(data)
data2 = run_query2(data)
!

data3 = process(data...
new* old style concurrency
def update_foo(data):
data1 = run_query1(data)
data2 = run_query2(data)
!

data3 = process(data...
new* old style concurrency
new* old style concurrency
new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
data1 = yield from run_query1(data)
data2 = yiel...
new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
data1 = yield from run_query1(data)
data2 = yiel...
new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
data1 = yield from run_query1(data)
data2 = yiel...
new* old style concurrency
new* old style concurrency
new* old style concurrency
new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
future1 = asyncio.async(run_query1(data))
future...
new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
future1 = asyncio.async(run_query1(data))
future...
new* old style concurrency
@asyncio.coroutine
def update_foo(data, loop):
future1 = asyncio.async(run_query1(data))
future...
new* old style concurrency
new* old style concurrency
new* old style concurrency
new* old style concurrency
def api_calls():
data1 = call_api1()
data2 = call_api2()
data3 = call_api3()
!

return [data1, ...
new* old style concurrency
def api_calls():
data1 = call_api1()
data3 = call_api2()
data3 = call_api3()
!

return [data1, ...
new* old style concurrency
new* old style concurrency
new* old style concurrency
@asyncio.coroutine
def api_calls():
future1 = asyncio.Task(call_api1())
future2 = asyncio.Task(...
new* old style concurrency
@asyncio.coroutine
def api_calls():
future1 = asyncio.Task(call_api1())
future2 = asyncio.Task(...
new* old style concurrency
new* old style concurrency
new* old style concurrency
new* old style concurrency
from aiohttp import request
!

@asyncio.coroutine
def call_api():
url = "URL"
response = yield ...
Takeaways
•

concurrency patterns

•

asyncio bootstrap

•

early adopters
Links
•

Video by BDFL Tulip: Async I/O for Python 3

•

Video by BDFL PyCon US 2013 Keynote

•

Code by Nikolay Kim aioht...
Upcoming SlideShare
Loading in...5
×

Python Async IO Horizon

1,541

Published on

~10min dive to Python Asynchronous IO

HTML version (recommended): https://dl.dropboxusercontent.com/u/1565687/speak/Python3%20AsyncIO%20Horizon/index.html

Published in: Technology
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,541
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
25
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide

Python Async IO Horizon

  1. 1. 19 Dec 2013 @lukmdo Python Asynchronous I/O asynchronous landscape refresher Photo by Jake Brown used under CC BY 2.0/“Mono+Balance”
  2. 2. Agenda ⏰10 minutes ! • Background (can skip) • Problem statement • “Current state” • Python 3 async horizon (by examples)
  3. 3. Warmup ! • CPU IO (CPU bound) • Pure IO (IO bound) • C10k • Concurrency vs. parallelism
  4. 4. Warmup ! • Lazy schedule a task for later • Avoid waiting for IO (i.e. 2x DB queries) • Solve embarrassingly parallel problems • Control and manage status of task
  5. 5. PEP index brief PEP Name Status Doc PEP 255 Simple Generators 2.2 ✔ »» PEP 342 Coroutines via Enhanced Generators 2.5 ✔ » PEP 380 Syntax for Delegating to a Subgenerator 3.3 ✔ » PEP 3148 Futures - execute computations asynchronously 3.2 ✔ » PEP 3153 Asynchronous IO support 3.3 ✔ » PEP 3156 Asynchronous IO Support Rebooted: the "asyncio" Module 3.4 ✔ »
  6. 6. PEP index brief PEP Name PEP 219 Stackless Python ❙❙ PEP 220 Coroutines, Generators, Continuations ✗ PEP 319 Python Synchronize/Asynchronize Block ✗ PEP 3145 PEP 3152 Asynchronous I/O For subprocess.Popen Cofunctions Status ❙❙ ❙❙
  7. 7. stackless In the Class concurrent.futures threading tornado greenlet subpocess eventlet celery twisted pyuv gevent shed asyncio asyncore
  8. 8. In the Class asyncio
  9. 9. stackless In the Class concurrent.futures threading tornado greenlet subpocess eventlet celery twisted pyuv gevent shed asyncio asyncore
  10. 10. In the Class concurrent.futures threading tornado subpocess celery twisted asyncio
  11. 11. Slang* • future ~ result wrapper • coro != generator function (but close) • task = coro wrapped in Future • result = yield from future/coro (to wait for result)
  12. 12. new* old style concurrency def update_foo(data): data1 = run_query1(data) data2 = run_query2(data) ! data3 = process(data1, data2) run_update_query(data3) run_update_cache(data3) ! return ["OK"]
  13. 13. new* old style concurrency def update_foo(data): data1 = run_query1(data) data2 = run_query2(data) ! data3 = process(data1, data2) run_update_query(data3) run_update_cache(data3) ! return ["OK"]
  14. 14. new* old style concurrency
  15. 15. new* old style concurrency
  16. 16. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): data1 = yield from run_query1(data) data2 = yield from run_query2(data) ! data3 = yield from process(data1, data2) yield from run_update_query(data3) loop.call_soon(update_cache, data3) ! return ["OK"]
  17. 17. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): data1 = yield from run_query1(data) data2 = yield from run_query2(data) ! data3 = yield from process(data1, data2) yield from run_update_query(data3) loop.call_soon(update_cache, data3) ! return ["OK"]
  18. 18. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): data1 = yield from run_query1(data) data2 = yield from run_query2(data) ! data3 = yield from process(data1, data2) yield from run_update_query(data3) loop.call_soon(update_cache, data3) ! return ["OK"]
  19. 19. new* old style concurrency
  20. 20. new* old style concurrency
  21. 21. new* old style concurrency
  22. 22. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): future1 = asyncio.async(run_query1(data)) future2 = asyncio.async(run_query2(data)) ! results = yield from asyncio.gather( future1, future2) data3 = yield from process(*results) yield from run_update_query(data3) ! loop.call_soon(update_cache, data3) ! return ["OK"]
  23. 23. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): future1 = asyncio.async(run_query1(data)) future2 = asyncio.async(run_query2(data)) ! results = yield from asyncio.gather( future1, future2) data3 = yield from process(*results) yield from run_update_query(data3) ! loop.call_soon(update_cache, data3) ! return ["OK"]
  24. 24. new* old style concurrency @asyncio.coroutine def update_foo(data, loop): future1 = asyncio.async(run_query1(data)) future2 = asyncio.async(run_query2(data)) ! results = yield from asyncio.gather( future1, future2) data3 = yield from process(*results) yield from run_update_query(data3) ! loop.call_soon(update_cache, data3) ! return ["OK"]
  25. 25. new* old style concurrency
  26. 26. new* old style concurrency
  27. 27. new* old style concurrency
  28. 28. new* old style concurrency def api_calls(): data1 = call_api1() data2 = call_api2() data3 = call_api3() ! return [data1, data2, data3]
  29. 29. new* old style concurrency def api_calls(): data1 = call_api1() data3 = call_api2() data3 = call_api3() ! return [data1, data2, data3]
  30. 30. new* old style concurrency
  31. 31. new* old style concurrency
  32. 32. new* old style concurrency @asyncio.coroutine def api_calls(): future1 = asyncio.Task(call_api1()) future2 = asyncio.Task(call_api2()) future3 = asyncio.Task(call_api3()) ! return ( yield from asyncio.gather( future1, future2, future3) )
  33. 33. new* old style concurrency @asyncio.coroutine def api_calls(): future1 = asyncio.Task(call_api1()) future2 = asyncio.Task(call_api2()) future3 = asyncio.Task(call_api3()) ! return ( yield from asyncio.gather( future1, future2, future3) )
  34. 34. new* old style concurrency
  35. 35. new* old style concurrency
  36. 36. new* old style concurrency
  37. 37. new* old style concurrency from aiohttp import request ! @asyncio.coroutine def call_api(): url = "URL" response = yield from request('GET', url) return (yield from response.read())
  38. 38. Takeaways • concurrency patterns • asyncio bootstrap • early adopters
  39. 39. Links • Video by BDFL Tulip: Async I/O for Python 3 • Video by BDFL PyCon US 2013 Keynote • Code by Nikolay Kim aiohttp • Code by google appengine-pipeline • Code by google ndb tasklets Photo by Jake Brown used under CC BY 2.0/“Mono+Tile”
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×