Python has been adding more and more async features to the language and the standard library. Starting with asyncio in python 3.4 and including the new async/await keywords in python 3.5, it’s difficult to understand how all these pieces fit together. More importantly, it’s hard to envision how to use these new language features in a real world application. In this talk we’re going to move beyond the basic examples of TCP echo servers and example servers that can add number together. Instead I’ll show you a realistic asyncio application. This application is a port of redis, a popular data structure server, written in python using asyncio. In addition to basic topics such as handling simple redis commands (GET, SET, RPUSH, etc), we’ll look at notifications using pub/sub, and how to implement blocking queues.
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
Writing Redis in Python with asyncio
1. WRITING REDIS IN PYTHON
WITH ASYNCIO
James Saryerwinnie / @jsaryer
2. GOALS
‣ How to structure a “larger” network server application
‣ Request/Response structure
‣ Publish/Subscribe
‣ Blocking queues
3. ABOUT ME
‣ AWS CLI
‣ Boto3/botocore/boto
‣ JMESPath
‣ AWS Shell
‣ Chalice
4. s Really Work (version 1.0) Create your own cartoon at www.projectc
customer
ned it
How the project leader
understood it
How the analyst designed
it
How the programmer
wrote it
How the business
consultant described it
http://www.projectcartoon.com/cartoon/3
How the authors envisioned it
How Projects Really Work (version 1.0)
How the customer
explained it
How the project leader
understood it
How the analyst designed
it
How
version 1.0) Create your own cartoon at www.projectcartoon.com
w the project leader
understood it
How the analyst designed
it
How the programmer
wrote it
How the business
consultant described it
What might happen here
5. REDIS
‣ Data structure server
‣ Set and get key value pairs
‣ Values can be more than string
36. rserver/db.py
_DB = {}
class DB:
def __init__(self, db=None):
if db is None:
db = _DB
self._db = db
def get(self, item):
return self._db.get(item)
def set(self, item, value):
self._db[item] = value
return True
‣ DB is in its own separate module
‣ It doesn’t know anything about asyncio
37. rserver/db.py
class DB:
def rpush(self, item, values):
current_list = self._db.setdefault(item, [])
current_list.extend(values)
return len(current_list)
def lrange(self, key, start, stop):
if stop == -1:
end = None
else:
stop += 1
return self._db.get(key, [])[start:stop]
def lpop(self, key):
value = self._db.get(key, [])
if value:
return value.pop(0)
60. rserver/db.py
from rserver import types
class DB:
def blpop(self, key):
value = self._db.get(key, [])
if value:
element = value.pop(0)
return element
return types.MUST_WAIT
61. rserver/db.py
from rserver import types
class DB:
def blpop(self, key):
value = self._db.get(key, [])
if value:
element = value.pop(0)
return element
return types.MUST_WAIT
62. rserver/db.py
from rserver import types
class DB:
def blpop(self, key):
value = self._db.get(key, [])
if value:
element = value.pop(0)
return element
return types.MUST_WAIT
63. rserver/protocol.py
class RedisServerProtocol(asyncio.Protocol):
def __init__(self, db, keyblocker, loop):
self._db = db
self._keyblocker = keyblocker
self._loop = loop
def data_received(self, data):
# …
if command == b'blpop':
response = self._db.blpop(
parsed[1], timeout=int(parsed[2]))
if response is types.MUST_WAIT:
q = self._keyblocker.wait_for_key(parsed[1],
self.transport)
self._loop.create_task(q)
return
64. class RedisServerProtocol(asyncio.Protocol):
def __init__(self, db, keyblocker, loop):
self._db = db
self._keyblocker = keyblocker
self._loop = loop
def data_received(self, data):
# …
if command == b'blpop':
response = self._db.blpop(
parsed[1], timeout=int(parsed[2]))
if response is types.MUST_WAIT:
q = self._keyblocker.wait_for_key(parsed[1],
self.transport)
self._loop.create_task(q)
return
rserver/protocol.py
65. class RedisServerProtocol(asyncio.Protocol):
def __init__(self, db, keyblocker, loop):
self._db = db
self._keyblocker = keyblocker
self._loop = loop
def data_received(self, data):
# …
if command == b'blpop':
response = self._db.blpop(
parsed[1], timeout=int(parsed[2]))
if response is types.MUST_WAIT:
q = self._keyblocker.wait_for_key(parsed[1],
self.transport)
self._loop.create_task(q)
return
rserver/protocol.py
66. class RedisServerProtocol(asyncio.Protocol):
def __init__(self, db, keyblocker, loop):
self._db = db
self._keyblocker = keyblocker
self._loop = loop
def data_received(self, data):
# …
if command == b'blpop':
response = self._db.blpop(
parsed[1], timeout=int(parsed[2]))
if response is types.MUST_WAIT:
q = self._keyblocker.wait_for_key(parsed[1],
self.transport)
self._loop.create_task(q)
return
rserver/protocol.py
67. class RedisServerProtocol(asyncio.Protocol):
def __init__(self, db, keyblocker, loop):
self._db = db
self._keyblocker = keyblocker
self._loop = loop
def data_received(self, data):
# …
if command == b'blpop':
response = self._db.blpop(
parsed[1], timeout=int(parsed[2]))
if response is types.MUST_WAIT:
q = self._keyblocker.wait_for_key(parsed[1],
self.transport)
self._loop.create_task(q)
return
rserver/protocol.py
89. rserver/server.py
class KeyBlocker:
def __init__(self):
self._blocked_keys = {}
async def wait_for_key(self, key, transport):
if key not in self._blocked_keys:
self._blocked_keys[key] = asyncio.Queue()
q = self._blocked_keys[key]
value = await q.get()
transport.write(
serializer.serialize_to_wire(value)
)
90. ADDITIONAL CONSIDERATIONS
‣ “Real” parsing is more complicated
‣ Pub/sub handles clients disconnecting
‣ Pub/sub globs
‣ Blocking queues can wait on multiple keys
91. PERFORMANCE
‣ redis-benchmark -n 100000 -t set,get -c 50
‣ redis-server: 82563 requests per second (gets/sets)
‣ pyredis-server: 24192 requests per second
‣ pyredis-server (uvloop): 38285 requests per second
92. WHAT WE LEARNED
‣ Transports and Protocols
‣ Simple request response
‣ Publish / Subscribe
‣ Blocking queue like behavior