Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Syncing up with Python’s asyncio for (micro) service development, Joir-dan Gumbs

1,160 views

Published on

PyParis 2017
http://pyparis.org

Published in: Technology
  • Be the first to comment

Syncing up with Python’s asyncio for (micro) service development, Joir-dan Gumbs

  1. 1. Syncing up with asyncio for (micro)service development Joir-dan Gumbs - Senior Software Engineer IBM San Francisco twitter: @jagumbs blog: tasteandtrace.co
  2. 2. what is asyncio?
  3. 3. what it isn’t
  4. 4. aioamqp aiobotocore aiocache aiocassandra aiocouchdb aiodns aiodocker aioes aioetcd aiofiles aiographite aiohttp aiokafka aiomcache aiomysql aioodbc aiopg aioprocessing aioredis aioredlock anoriak aiosmtpd aiotasks aiozk aiozmq async_timeout motor prometheus_async rethinkdb sanic uvloop https://github.com/python/asyncio/wiki/ThirdParty quite a few asyncio-ready packages exist already
  5. 5. building (micro)services
  6. 6. aiohttp asyncio web framework sanic flask-like web server
  7. 7. async keyword is an explicit indicator of asynchronous construct async def - Indicates coroutine (asynchronous function) async with - Indicates asynchronous context manager async for - Indicates asynchronous iteration async + await + event loop = asyncio
  8. 8. await keyword is an explicit asynchronous “checkpoint” Signal to swap to different coroutine Will resume from checkpoint when task completed and has regained context async + await + event loop = asyncio
  9. 9. event loop is the main execution device for asyncio handles registration, execution, and cancelling of coroutines expensive synchronous calls will block the event loop until completion can delegate expensive (IO/CPU) calls to various Executors async + await + event loop = asyncio
  10. 10. hello asyncio 1 - simple coroutine import asyncio import random async def work_it(i, random_sleep): await asyncio.sleep(random_sleep) print("Task {0} completed after random sleep of {1} seconds".format(i, random_sleep)) if __name__ == "__main__": print("Run basic async function.") loop = asyncio.get_event_loop() loop.run_until_complete(work_it(1, random.randrange(1, 5))) print("Finished!")
  11. 11. hello sanic 1 - hello_sanic.py import asyncio import uvloop from sanic import Sanic from sanic.response import text app = Sanic(“hello_sanic”) @app.route(“/async”) async def async_hello(request): return text(“Asynchronous Hello Sanic!”) @app.route(“/sync”) def sync_hello(request): return text(“Synchronous Hello Sanic!”) if __name__ == "__main__": # Use uvloop cause its fast! asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) app.run(host="0.0.0.0", port=9090)
  12. 12. coroutines return a Future object using await unwraps Future for value(s), if available we can await multiple Future objects using asyncio.gather callbacks are optional with asyncio coroutines created using <Future>.add_done_callback, if you want them async + await + event loop = asyncio
  13. 13. hello asyncio 2 - coroutine collection import asyncio import random async def work_it(i, random_sleep): await asyncio.sleep(random_sleep) print("Task {0} completed after random sleep of {1} seconds".format(i, random_sleep)) if __name__ == "__main__": print("Run basic async function with grouping.") loop = asyncio.get_event_loop() futures = [work_it(i, random.randrange(1, 5)) for i in range(0, 10)] group_future = asyncio.gather(*futures) loop.run_until_complete(group_future) print("Finished!")
  14. 14. hello sanic 2 - outbound.py import asyncio import aiohttp import uvloop from sanic import Sanic from sanic.response import json app = Sanic(“hello_sanic”) @app.route(“/”) async def hello(request): site = “http://www.ibm.com" response = await aiohttp.request(“GET”, site) return json({“website”: site, “size”: len(await response.text())}) if __name__ == "__main__": # Use uvloop cause its fast! asyncio.set_event_loop_policy(uvloop.EventLoopPolicy()) app.run(host="0.0.0.0", port=9090)
  15. 15. executors help us use synchronous functions asynchronously <event_loop>.run_in_executor(executor, fn, *args) is our friend! use functools.partial if you require **kwargs works with concurrent.futures.[ThreadPoolExecutor, ProcessPoolExecutor] defaults to ThreadPoolExecutor when None asynci-oh no! - executors to the rescue
  16. 16. asynci-oh no! - a sync in my async import asyncio import requests async def run(loop): print("Accessing ibm.com”) # requests is a synchronous http framework. . . result = requests.get(“http://www.ibm.com”) # I’m a blocker!! print("Status Code", result.status_code) print("Body Length", len(result.text)) if __name__ == "__main__": print("Run basic async function with executor.") loop = asyncio.get_event_loop() loop.run_until_complete(run(loop)) print("Finished!")
  17. 17. hello asyncio 3 - async a sync import asyncio import requests async def run(loop): print("Accessing ibm.com”) # There are better ways of doing this though . . . result = await loop.run_in_executor(None, requests.get, "http://www.ibm.com") print("Status Code", result.status_code) print("Body Length", len(result.text)) if __name__ == "__main__": print("Run basic async function with executor.") loop = asyncio.get_event_loop() loop.run_until_complete(run(loop)) print("Finished!")
  18. 18. hello asyncio 3 - async a sync import asyncio import requests async def run(loop): print("Accessing ibm.com”) # There are better ways of doing this though . . . result = await loop.run_in_executor(None, requests.get, "http://www.ibm.com") print("Status Code", result.status_code) print("Body Length", len(result.text)) if __name__ == "__main__": print("Run basic async function with executor.") loop = asyncio.get_event_loop() loop.run_until_complete(run(loop)) print("Finished!")
  19. 19. hello asyncio 3 - async a sync (aiohttp edition) import aiohttp import asyncio async def run(): print("Accessing ibm.com") result = await aiohttp.request("GET", "http://www.ibm.com") print("Status Code", result.status) print("Body Length", len(await result.text())) if __name__ == "__main__": print("Run basic async function with aiohttp.") loop = asyncio.get_event_loop() loop.run_until_complete(run()) print("Finished!")
  20. 20. hello asyncio 6 - amethyst Go to code https://github.com/SakuraSound/amethyst
  21. 21. Similar to standard python context manager __aenter__ and __aexit__ vs. __enter__ and __exit__ can wait on I/O while creating/destroying “session” async with - establishing “sessions”
  22. 22. hello asyncio 4 - PlayerState context manager definition class PlayerState(): . . . def __init__(self, player_id, region='US'): self.loop = asyncio.get_event_loop() self.player_id = player_id self.region = region async def __aenter__(self): self.lock = await self.get_lock(self.player_id) self.state = await self.loop.run_in_executor(None, self.get_state, self.player_id) print("locking {0}".format(self.player_id)) return self async def __aexit__(self, exc_type, exc, tb): if hasattr(self, 'lock'): if hasattr(self, 'state'): await self.loop.run_in_executor(None, self.push_state) await self.unlock() else: raise AssertionError("Did we have the lock?”) . . .
  23. 23. hello asyncio 4 - PlayerState context manager usage . . . async def run(player): async with PlayerState(player) as ps: await ps.update("foo", "bar") print(ps.state) if __name__ == "__main__": loop = asyncio.get_event_loop() loop.run_until_complete(run("SakuraSound")) . . .
  24. 24. Looping construct for handling iterators and generators iterators must implement __aiter__ and __anext__ async for - iterate and generate
  25. 25. hello asyncio 5 - async_for_iterator.py import asyncio import random class NextSweets(object): __sprites__ = ["bonbon", "taffy", "mochi", "cookie", "cake"] def __init__(self, to): self.curr = 0 self.to = to async def __anext__(self): if self.curr >= self.to: raise StopAsyncIteration self.curr += 1 i = random.randint(0, len(NextSweets.__sprites__) - 1) await asyncio.sleep(1) return NextSweets.__sprites__[i] async def __aiter__(self): return self . . .
  26. 26. hello asyncio 5 - async_for_iterator.py . . . @classmethod async def get_next_k(cls, k): async for sweet in NextSweets(k): print(sweet) loop = asyncio.get_event_loop() loop.run_until_complete(NextSweets.get_next_k(30))
  27. 27. hello asyncio 6 - async_for_generator.py import asyncio import random async def next_k_sweets(k, sweets): for i in range(k): # Simulating blocking call (DB? Service?) await asyncio.sleep(1) pos = random.randint(0, len(sweets) - 1) yield sweets[pos] async def run(): sweets = ["bonbon", "taffy", "mochi", "cookie", "cake", "ice_cream"] async for sweet in next_k_sweets(50, sweets): print(sweet)
  28. 28. hello asyncio 6 - starcraft_replay_processing Go to code https://github.com/SakuraSound/replay_parsing
  29. 29. not everything has to be a coroutine!
  30. 30. try to use asyncio packages when available...
  31. 31. ...and use executors if they aren’t.
  32. 32. questions?

×