Python Async IO Horizon
Upcoming SlideShare
Loading in...5
×
 

Python Async IO Horizon

on

  • 912 views

~10min dive to Python Asynchronous IO

~10min dive to Python Asynchronous IO

HTML version (recommended): https://dl.dropboxusercontent.com/u/1565687/speak/Python3%20AsyncIO%20Horizon/index.html

Statistics

Views

Total Views
912
Views on SlideShare
912
Embed Views
0

Actions

Likes
2
Downloads
14
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Python Async IO Horizon Python Async IO Horizon Presentation Transcript

  • 19 Dec 2013 @lukmdo Python Asynchronous I/O asynchronous landscape refresher Photo by Jake Brown used under CC BY 2.0/“Mono+Balance”
  • Agenda ⏰10 minutes ! • Background (can skip) • Problem statement • “Current state” • Python 3 async horizon (by examples)
  • Warmup ! • CPU IO (CPU bound) • Pure IO (IO bound) • C10k • Concurrency vs. parallelism
  • Warmup ! • Lazy schedule a task for later • Avoid waiting for IO (i.e. 2x DB queries) • Solve embarrassingly parallel problems • Control and manage status of task
  • PEP index brief PEP Name Status Doc PEP 255 Simple Generators 2.2 ✔ »» PEP 342 Coroutines via Enhanced Generators 2.5 ✔ » PEP 380 Syntax for Delegating to a Subgenerator 3.3 ✔ » PEP 3148 Futures - execute computations asynchronously 3.2 ✔ » PEP 3153 Asynchronous IO support 3.3 ✔ » PEP 3156 Asynchronous IO Support Rebooted: the "asyncio" Module 3.4 ✔ »
  • PEP index brief PEP Name PEP 219 Stackless Python ❙❙ PEP 220 Coroutines, Generators, Continuations ✗ PEP 319 Python Synchronize/Asynchronize Block ✗ PEP 3145 PEP 3152 Asynchronous I/O For subprocess.Popen Cofunctions Status ❙❙ ❙❙
  • stackless In the Class concurrent.futures threading tornado greenlet subpocess eventlet celery twisted pyuv gevent shed asyncio asyncore
  • In the Class asyncio
  • stackless In the Class concurrent.futures threading tornado greenlet subpocess eventlet celery twisted pyuv gevent shed asyncio asyncore
  • In the Class concurrent.futures threading tornado subpocess celery twisted asyncio
  • Slang* • future ~ result wrapper • coro != generator function (but close) • task = coro wrapped in Future • result = yield from future/coro (to wait for result)
  • new* old style concurrency def update_foo(data): data1 = run_query1(data) data2 = run_query2(data) ! data3 = process(data1, data2) run_update_query(data3) run_update_cache(data3) ! return ["OK"]
  • new* old style concurrency def update_foo(data): data1 = run_query1(data) data2 = run_query2(data) ! data3 = process(data1, data2) run_update_query(data3) run_update_cache(data3) ! return ["OK"]
  • new* old style concurrency
  • new* old style concurrency
  • new* old style concurrency @asyncio.coroutine def update_foo(data, loop): data1 = yield from run_query1(data) data2 = yield from run_query2(data) ! data3 = yield from process(data1, data2) yield from run_update_query(data3) loop.call_soon(update_cache, data3) ! return ["OK"]
  • new* old style concurrency @asyncio.coroutine def update_foo(data, loop): data1 = yield from run_query1(data) data2 = yield from run_query2(data) ! data3 = yield from process(data1, data2) yield from run_update_query(data3) loop.call_soon(update_cache, data3) ! return ["OK"]
  • new* old style concurrency @asyncio.coroutine def update_foo(data, loop): data1 = yield from run_query1(data) data2 = yield from run_query2(data) ! data3 = yield from process(data1, data2) yield from run_update_query(data3) loop.call_soon(update_cache, data3) ! return ["OK"]
  • new* old style concurrency
  • new* old style concurrency
  • new* old style concurrency
  • new* old style concurrency @asyncio.coroutine def update_foo(data, loop): future1 = asyncio.async(run_query1(data)) future2 = asyncio.async(run_query2(data)) ! results = yield from asyncio.gather( future1, future2) data3 = yield from process(*results) yield from run_update_query(data3) ! loop.call_soon(update_cache, data3) ! return ["OK"]
  • new* old style concurrency @asyncio.coroutine def update_foo(data, loop): future1 = asyncio.async(run_query1(data)) future2 = asyncio.async(run_query2(data)) ! results = yield from asyncio.gather( future1, future2) data3 = yield from process(*results) yield from run_update_query(data3) ! loop.call_soon(update_cache, data3) ! return ["OK"]
  • new* old style concurrency @asyncio.coroutine def update_foo(data, loop): future1 = asyncio.async(run_query1(data)) future2 = asyncio.async(run_query2(data)) ! results = yield from asyncio.gather( future1, future2) data3 = yield from process(*results) yield from run_update_query(data3) ! loop.call_soon(update_cache, data3) ! return ["OK"]
  • new* old style concurrency
  • new* old style concurrency
  • new* old style concurrency
  • new* old style concurrency def api_calls(): data1 = call_api1() data2 = call_api2() data3 = call_api3() ! return [data1, data2, data3]
  • new* old style concurrency def api_calls(): data1 = call_api1() data3 = call_api2() data3 = call_api3() ! return [data1, data2, data3]
  • new* old style concurrency
  • new* old style concurrency
  • new* old style concurrency @asyncio.coroutine def api_calls(): future1 = asyncio.Task(call_api1()) future2 = asyncio.Task(call_api2()) future3 = asyncio.Task(call_api3()) ! return ( yield from asyncio.gather( future1, future2, future3) )
  • new* old style concurrency @asyncio.coroutine def api_calls(): future1 = asyncio.Task(call_api1()) future2 = asyncio.Task(call_api2()) future3 = asyncio.Task(call_api3()) ! return ( yield from asyncio.gather( future1, future2, future3) )
  • new* old style concurrency
  • new* old style concurrency
  • new* old style concurrency
  • new* old style concurrency from aiohttp import request ! @asyncio.coroutine def call_api(): url = "URL" response = yield from request('GET', url) return (yield from response.read())
  • Takeaways • concurrency patterns • asyncio bootstrap • early adopters
  • Links • Video by BDFL Tulip: Async I/O for Python 3 • Video by BDFL PyCon US 2013 Keynote • Code by Nikolay Kim aiohttp • Code by google appengine-pipeline • Code by google ndb tasklets Photo by Jake Brown used under CC BY 2.0/“Mono+Tile”