Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Асинхронный RPC с
помощью Gevent и RabbitMQ
Александр Мокров
О чем доклад
Некоторые ограничения Celery
Как их обойти
Gevent
RabbitMQ (некоторые особенности)
Будет предложена модель ас...
Пример построения приложения на Celery workflow
get flour
bake pie
get meat
seal pie
create
dough
order pie
get milk
get a...
...
entry point
2
entry point
1
entry point
3 service n
app
service 2
service 1
sommelier
winery
сhateau
Service of
degustation 3
app
Service of
degustation 2
Service of
degustation 1
Дегустация вин с
gevent и RabbitMQ
entry task
service task 1
DB
callback task 1
service task n
callback task n
Что хотелось бы получить
entry task
service task 1
service task 2
service task n
long task services
time
Celery AsyncResult
async_result = task.apply_async()
print(async_result.status)
False
result = async_result.wait()
long task
service
task
persistant
queue
exclusive
queue
reply_to=amq.gen-E6.
correlation_id
Request
correlation_id
Respons...
greenlet
greenlet
long task
service
response
listener
service service service
reply_to
exclusive
queue
services queues
Workers
● solo
● prefork
● eventlet
● gevent
Gevent
gevent is a concurrency library based around libev. It provides a clean API for a
variety of concurrency and networ...
Greenlet
The primary pattern used in gevent is the Greenlet, a lightweight coroutine
provided to Python as a C extension m...
Event loop
def foo():
print('Running in foo')
gevent.sleep(0)
print('Explicit context switch to
foo')
def bar():
print('Explicit cont...
def task(pid):
gevent.sleep(random.randint(0,2)*0.001)
print('Task %s done' % pid)
def synchronous():
for i in range(1, 8)...
def echo(i):
time.sleep(0.001)
return i
# Non Deterministic Process Pool
from multiprocessing.pool import Pool
p = Pool(10...
# Deterministic Gevent Pool
from gevent.pool import Pool
p = Pool(10)
run1 = [a for a in p.imap_unordered(echo, xrange(10)...
Spawning Greenlets
from gevent import Greenlet
thread1 = Greenlet.spawn(foo, "message", 1)
thread2 = gevent.spawn(foo, "me...
class MyGreenlet(Greenlet):
def __init__(self, message, n):
Greenlet.__init__(self)
self.message = message
self.n = n
def ...
Greenlet State
started -- Boolean, indicates whether the Greenlet has been started
ready() -- Boolean, indicates whether t...
greenlet
greenlet
long task
service
response
listener
subscribe(task_id)
Timeouts
from gevent import Timeout
seconds = 10
timeout = Timeout(seconds)
timeout.start()
def wait():
gevent.sleep(10)
try:
geven...
time_to_wait = 5 # seconds
class TooLong(Exception):
pass
with Timeout(time_to_wait, TooLong):
gevent.sleep(10)
class Queue(maxsize=None, items=None)
empty()
full()
get(block=True, timeout=None)
get_nowait()
next()
peek(block=True, ti...
greenlet
greenlet
greenlet
service
results
dispatcher
gevent.queues
task_id
reply_to, results_queue
Events
Groups and
Pools
Locks and
Semaphores
Subprocess
Thread
Locals
Actors
Monkey patching
guerrilla
patch
gorilla
patch
monkey
patch
import socket
print(socket.socket)
from gevent import monkey
monkey.patch_socket()
print("After monkey patch")
print(socke...
Stack layout for a greenlet
| ^^^ |
| older data |
| |
stack_stop . |_______________|
. | |
. | greenlet data |
. | in sta...
greenlet
greenlet
greenlet
service
results
dispatcher
service service service
reply_to
exclusive
queue
services queues
sub...
Service Result Dispatcher
greenlet
greenlet
greenlet
service
results
dispatcher
reply_to
exclusive
queue
reply_to, results_queue
gevent.queues
task_...
class ServiceResultsDispatcher(Greenlet):
def __init__(self):
…
self.reply_to = None
Greenlet.__init__(self)
def create_co...
def subscribe(self, task_id):
service_results_queue = gevent.queue.Queue()
self.service_results[task_id] = service_results...
def _run(self):
while True:
try:
for method_frame, properties, body in self.channel.consume(self.reply_to,
no_ack=True):
i...
Greenlet task
greenlet
greenlet
greenlet
service
results
dispatcher
services queues
subscribe
gevent.queues
self.results_queue, self.reply_to = self.service_publisher.subscribe(self.task_id)
self.channel.basic_publish(exchange='',...
try:
method_frame, properties, body =
self.results_queue.get(block=True, timeout=self.timeout)
except Empty:
logger.info('...
Services
response = channel.basic_publish(
exchange='',
routing_key=props.reply_to,
properties=BasicProperties(correlation...
Альтернативы?
Почему gevent?
1. Встроенная поддержка в Celery (малыми силами)
2. Хотелось рассмотреть в докладе именно gev...
Вывод
Ссылки
http://www.gevent.org
http://sdiehl.github.io/gevent-tutorial/
https://github.com/python-greenlet/greenlet
https://...
Спасибо за внимание!
Gevent rabbit rpc
Upcoming SlideShare
Loading in …5
×

Gevent rabbit rpc

211 views

Published on

Gevent RabbitMQ Async RPC

Published in: Software
  • Be the first to comment

  • Be the first to like this

Gevent rabbit rpc

  1. 1. Асинхронный RPC с помощью Gevent и RabbitMQ Александр Мокров
  2. 2. О чем доклад Некоторые ограничения Celery Как их обойти Gevent RabbitMQ (некоторые особенности) Будет предложена модель асинхронного RPC
  3. 3. Пример построения приложения на Celery workflow get flour bake pie get meat seal pie create dough order pie get milk get aggs
  4. 4. ... entry point 2 entry point 1 entry point 3 service n app service 2 service 1
  5. 5. sommelier winery сhateau Service of degustation 3 app Service of degustation 2 Service of degustation 1
  6. 6. Дегустация вин с gevent и RabbitMQ
  7. 7. entry task service task 1 DB callback task 1 service task n callback task n
  8. 8. Что хотелось бы получить entry task service task 1 service task 2 service task n
  9. 9. long task services time
  10. 10. Celery AsyncResult async_result = task.apply_async() print(async_result.status) False result = async_result.wait()
  11. 11. long task service task persistant queue exclusive queue reply_to=amq.gen-E6. correlation_id Request correlation_id Response reply_to=amq.gen-E6... RabbitMQ RPC
  12. 12. greenlet greenlet long task service response listener service service service reply_to exclusive queue services queues
  13. 13. Workers ● solo ● prefork ● eventlet ● gevent
  14. 14. Gevent gevent is a concurrency library based around libev. It provides a clean API for a variety of concurrency and network related tasks.
  15. 15. Greenlet The primary pattern used in gevent is the Greenlet, a lightweight coroutine provided to Python as a C extension module. Greenlets all run inside of the OS process for the main program but are scheduled cooperatively. Only one greenlet is ever running at any given time. Spin-off of Stackless, a version of CPython that supports micro-threads called “tasklets”. Tasklets run pseudo-concurrently (typically in a single or a few OS-level threads) and are synchronized with data exchanges on “channels”. Its coroutine
  16. 16. Event loop
  17. 17. def foo(): print('Running in foo') gevent.sleep(0) print('Explicit context switch to foo') def bar(): print('Explicit context to bar') gevent.sleep() print('Implicit context switch to bar') gevent.joinall([ gevent.spawn(foo), gevent.spawn(bar), ]) Running in foo Explicit context to bar Explicit context switch to foo Implicit context switch to bar
  18. 18. def task(pid): gevent.sleep(random.randint(0,2)*0.001) print('Task %s done' % pid) def synchronous(): for i in range(1, 8): task(i) def asynchronous(): threads = [gevent.spawn(task, i) for i in range(10)] gevent.joinall(threads) Synchronous: Task 1 done Task 2 done Task 3 done Task 4 done Task 5 done Task 6 done Task 7 done Asynchronous: Task 1 done Task 5 done Task 6 done Task 2 done Task 4 done Task 7 done Task 0 done Task 3 done
  19. 19. def echo(i): time.sleep(0.001) return i # Non Deterministic Process Pool from multiprocessing.pool import Pool p = Pool(10) run1 = [a for a in p.imap_unordered(echo, xrange(10))] run2 = [a for a in p.imap_unordered(echo, xrange(10))] run3 = [a for a in p.imap_unordered(echo, xrange(10))] run4 = [a for a in p.imap_unordered(echo, xrange(10))] print(run1 == run2 == run3 == run4) False
  20. 20. # Deterministic Gevent Pool from gevent.pool import Pool p = Pool(10) run1 = [a for a in p.imap_unordered(echo, xrange(10))] run2 = [a for a in p.imap_unordered(echo, xrange(10))] run3 = [a for a in p.imap_unordered(echo, xrange(10))] run4 = [a for a in p.imap_unordered(echo, xrange(10))] print(run1 == run2 == run3 == run4) True
  21. 21. Spawning Greenlets from gevent import Greenlet thread1 = Greenlet.spawn(foo, "message", 1) thread2 = gevent.spawn(foo, "message", 2) thread3 = gevent.spawn(lambda x: (x+1), 2) threads = [thread1, thread2, thread3] # Block until all threads complete. gevent.joinall(threads)
  22. 22. class MyGreenlet(Greenlet): def __init__(self, message, n): Greenlet.__init__(self) self.message = message self.n = n def _run(self): print(self.message) gevent.sleep(self.n) g = MyGreenlet("Hi there!", 3) g.start() g.join()
  23. 23. Greenlet State started -- Boolean, indicates whether the Greenlet has been started ready() -- Boolean, indicates whether the Greenlet has halted successful() -- Boolean, indicates whether the Greenlet has halted and not thrown an exception value -- arbitrary, the value returned by the Greenlet exception -- exception, uncaught exception instance thrown inside the greenlet
  24. 24. greenlet greenlet long task service response listener subscribe(task_id)
  25. 25. Timeouts
  26. 26. from gevent import Timeout seconds = 10 timeout = Timeout(seconds) timeout.start() def wait(): gevent.sleep(10) try: gevent.spawn(wait).join() except Timeout: print('Could not complete')
  27. 27. time_to_wait = 5 # seconds class TooLong(Exception): pass with Timeout(time_to_wait, TooLong): gevent.sleep(10)
  28. 28. class Queue(maxsize=None, items=None) empty() full() get(block=True, timeout=None) get_nowait() next() peek(block=True, timeout=None) peek_nowait() put(item, block=True, timeout=None) put_nowait(item) qsize()
  29. 29. greenlet greenlet greenlet service results dispatcher gevent.queues task_id reply_to, results_queue
  30. 30. Events Groups and Pools Locks and Semaphores Subprocess Thread Locals Actors
  31. 31. Monkey patching guerrilla patch gorilla patch monkey patch
  32. 32. import socket print(socket.socket) from gevent import monkey monkey.patch_socket() print("After monkey patch") print(socket.socket) import select print(select.select) monkey.patch_select() print("After monkey patch") print(select.select) <class 'socket.socket'> After monkey patch <class 'gevent._socket3.socket'> <built-in function select> After monkey patch <function select at 0x7ff7e111c378>
  33. 33. Stack layout for a greenlet | ^^^ | | older data | | | stack_stop . |_______________| . | | . | greenlet data | . | in stack | . * |_______________| . . _____________ stack_copy + stack_saved . | | | | . | data | |greenlet data| . | unrelated | | saved | . | to | | in heap | stack_start . | this | . . |_____________| stack_copy | greenlet | | | | newer data | | vvv |
  34. 34. greenlet greenlet greenlet service results dispatcher service service service reply_to exclusive queue services queues subscribe gevent.queues
  35. 35. Service Result Dispatcher
  36. 36. greenlet greenlet greenlet service results dispatcher reply_to exclusive queue reply_to, results_queue gevent.queues task_id task_id, reply_to
  37. 37. class ServiceResultsDispatcher(Greenlet): def __init__(self): … self.reply_to = None Greenlet.__init__(self) def create_connection(self): ... result = self.channel.queue_declare(exclusive=True) self.reply_to = result.method.queue
  38. 38. def subscribe(self, task_id): service_results_queue = gevent.queue.Queue() self.service_results[task_id] = service_results_queue return service_results, self.reply_to def unsubscribe(self, task_id): self.service_results.pop(task_id, None)
  39. 39. def _run(self): while True: try: for method_frame, properties, body in self.channel.consume(self.reply_to, no_ack=True): if properties.correlation_id in self.tasks: self.tasks[properties.correlation_id].put_nowait((method_frame, properties, body)) except ...
  40. 40. Greenlet task
  41. 41. greenlet greenlet greenlet service results dispatcher services queues subscribe gevent.queues
  42. 42. self.results_queue, self.reply_to = self.service_publisher.subscribe(self.task_id) self.channel.basic_publish(exchange='', routing_key=service_queue, properties=BasicProperties( reply_to=self.reply_to, correlation_id=self.task_id ), body=request)
  43. 43. try: method_frame, properties, body = self.results_queue.get(block=True, timeout=self.timeout) except Empty: logger.info('timeout') break else: logger.info('body = {}'.format(body))
  44. 44. Services response = channel.basic_publish( exchange='', routing_key=props.reply_to, properties=BasicProperties(correlation_id=request.task_id), body=response )
  45. 45. Альтернативы? Почему gevent? 1. Встроенная поддержка в Celery (малыми силами) 2. Хотелось рассмотреть в докладе именно gevent. Ничто не мешает переделать, к примеру, на asyncio.
  46. 46. Вывод
  47. 47. Ссылки http://www.gevent.org http://sdiehl.github.io/gevent-tutorial/ https://github.com/python-greenlet/greenlet https://www.rabbitmq.com/ http://www.celeryproject.org/
  48. 48. Спасибо за внимание!

×