Distributed RPC in Nova with ZeroMQ

9,563 views
9,356 views

Published on

Presentation deck from OpenStack Design Summit spring 2012 in San Francisco, by Eric Windisch.

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
9,563
On SlideShare
0
From Embeds
0
Number of Embeds
5,855
Actions
Shares
0
Downloads
124
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Distributed RPC in Nova with ZeroMQ

  1. 1. X Image: http://ken.pepple.info/openstack/2011/04/22/openstack-nova-architecture/ Distributed RPC in Nova with ZeroMQ 04-16-2012 Distributed RPC with ZeroMQ: code CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution & analysis 1 CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  2. 2. Etherpad http://etherpad.openstack.org/ FolsomDistributedRPC 04-16-2012 Distributed RPC with ZeroMQ: code CCA - NoDerivs 3.0 Unported License -2Usage OK, no modifications, full attribution & analysis CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  3. 3. @ewindisch 04-16-2012 Distributed RPC with ZeroMQ: code CCA - NoDerivs 3.0 Unported License -3Usage OK, no modifications, full attribution & analysis CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  4. 4. Etherpad http://etherpad.openstack.org/ FolsomDistributedRPC CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  5. 5. Application Message Queue Broker Brokered Messaging 04-16-2012 Distributed RPC with ZeroMQ: code CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution & analysis 5 CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  6. 6. Application X Message Queue Broker Brokered Messaging 04-16-2012 Distributed RPC with ZeroMQ: code CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution & analysis 5 CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  7. 7. Application Message Queue Brokered Messaging CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  8. 8. Application Brokered Messaging CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  9. 9. Application HA Failover High-Availability CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  10. 10. Application failover fails CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  11. 11. Nova Nova Nova API Network Compute Scheduler Message Message Queue Queue Nova Nova Nova API Network Compute Scheduler Application Decentralized CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  12. 12. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  13. 13. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  14. 14. Nova Nova Nova API Network Compute Scheduler Message Message Queue Queue Nova Nova Nova API Network Compute Scheduler Application Decentralized CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  15. 15. Nova Nova Nova API Network Compute Scheduler Message Abandoned Queue Messages Nova Nova Nova API Network Compute Scheduler Application Decentralized CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  16. 16. Nova Nova Nova API Network Compute Scheduler Message Queue Nova Nova Nova API Network Compute Scheduler Application Decentralized CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  17. 17. Solution: Distributed RPC for OpenStack Nova CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  18. 18. API Nova API Application Nova Nova Network Nova Scheduler Network Nova Nova Compute Compute Peer2Peer CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  19. 19. API Nova API Application Nova Nova Network Network Nova Nova Compute Compute Peer2Peer CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  20. 20. API Nova API Application Nova Scheduler Nova Nova Network Network Nova Scheduler Nova Nova Compute Compute Peer2Peer CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  21. 21. API Nova API Application Nova Nova Network Network Nova Scheduler Nova Nova Compute Compute Peer2Peer CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  22. 22. rpc.cast(topic.host) PUSH -> PULL Direct Publisher Topic Consumer Invoker Worker (e.g. api) (e.g. compute) Cast to host. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  23. 23. rpc.cast(topic.host) PUSH -> PULL Direct Publisher Topic Consumer Invoker Worker (e.g. api) (e.g. compute) Cast to host. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  24. 24. rpc.cast(topic.host) PUSH -> PULL Direct Publisher Topic Consumer Invoker Worker (e.g. api) (e.g. compute) Reply. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  25. 25. rpc.cast(topic.host) PUSH -> PULL Direct Publisher Topic Consumer Invoker Worker (e.g. api) (e.g. compute) Reply. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  26. 26. def _send(...): “”” code omitted “”” # Curry the original request into a reply method. orig_payload = [RpcContext.marshal(context), msg] payload = [RpcContext.marshal(context), { method: -reply, # Send original call to wrapper args: { msg_id: msg_id, context: RpcContext.marshal(context), topic: reply_topic, # our return path msg: orig_payload } }]Tuesday, April 17, 12
  27. 27. # Messages arriving async. msg_waiter = QueueSocket( "ipc:///var/run/nova/zmq_reply_queue", zmq.SUB, subscribe=msg_id, bind=False) try: with Timeout(timeout): with Timeout(30) as t1: conn.cast(msg_id, topic, payload) # Blocks until receives reply responses = pickle.loads(msg_waiter.recv()[-1])Tuesday, April 17, 12
  28. 28. def reply(...): # this is -reply """Reply to a casted call""" child_ctx = RpcContext.unmarshal(orig_payload[0]) response = ConsumerBase.normalize_reply( self.get_response( # execute original call child_ctx, proxy, topic, orig_payload[1]), ctx.replies ) _multi_send("cast", ctx, reply_topic, { method: -process_reply, args: { msg_id: msg_id, response: response } })Tuesday, April 17, 12
  29. 29. def process_reply(...): """Process a reply""" # zmq_replies is the receiver service self.msg_waiter.cast(str(msg_id), str(zmq_replies), response)Tuesday, April 17, 12
  30. 30. rpc.call(topic.worker) PUSH->PULL curries reply method Direct Publisher Topic Consumer PULL<-PUSH Invoker rpc.cast -process_reply() Worker (e.g. api) (e.g. compute.host) Call method. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  31. 31. rpc.call(topic.worker) PUSH->PULL curries reply method Direct Publisher Topic Consumer PULL<-PUSH Invoker rpc.cast -process_reply() Worker (e.g. api) (e.g. compute.host) Call method. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  32. 32. rpc.call(topic.worker) PUSH->PULL curries reply method Direct Publisher Topic Consumer PULL<-PUSH Invoker rpc.cast -process_reply() Worker (e.g. api) (e.g. compute.host) Call method. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  33. 33. rpc.call(topic.worker) PUSH->PULL curries reply method Direct Publisher Topic Consumer PULL<-PUSH Invoker rpc.cast -process_reply() Worker (e.g. api) (e.g. compute.host) Call method. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  34. 34. Bare Topics “scheduler.host1” goes to “host1” Where do messages for “scheduler” go? CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  35. 35. MatchMaking CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  36. 36. Rewrite Engine • Every incoming topic is mapped to an array of topics and hosts. • Usually returns a single topic/ host. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  37. 37. Search for hosts • Can be provided by: • hash table (implemented) • name service (i.e. DNS) • directory service (i.e. LDAP) • zookeeper • etc... CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  38. 38. rpc.fanout_cast(topic) MatchMaker e.g. "compute" Fanout may have many workers. get_workers as workers Array *topic.worker Direct for each Publisher Topic worker: Consumer PUSH -> PULL Invoker i.e. rpc.cast(topic.worker) Worker (e.g. api) (e.g. nova-compute on compute.host) Fanout Cast. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  39. 39. rpc.fanout_cast(topic) MatchMaker e.g. "compute" Fanout may have many workers. get_workers as workers Array *topic.worker Direct for each Publisher Topic worker: Consumer PUSH -> PULL Invoker i.e. rpc.cast(topic.worker) Worker (e.g. api) (e.g. nova-compute on compute.host) Fanout Cast. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  40. 40. rpc.fanout_cast(topic) MatchMaker e.g. "compute" Fanout may have many workers. get_workers as workers Array *topic.worker Direct for each Publisher Topic worker: Consumer PUSH -> PULL Invoker i.e. rpc.cast(topic.worker) Worker (e.g. api) (e.g. nova-compute on compute.host) Fanout Cast. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  41. 41. rpc.fanout_cast(topic) MatchMaker e.g. "compute" Fanout may have many workers. get_workers as workers Array *topic.worker Direct for each Publisher Topic worker: Consumer PUSH -> PULL Invoker i.e. rpc.cast(topic.worker) Worker (e.g. api) (e.g. nova-compute on compute.host) Fanout Cast. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  42. 42. rpc.fanout_cast(topic) MatchMaker e.g. "compute" Fanout may have many workers. get_workers as workers Array *topic.worker Direct for each Publisher Topic worker: Consumer PUSH -> PULL Invoker i.e. rpc.cast(topic.worker) Worker (e.g. api) (e.g. nova-compute on compute.host) Fanout Cast. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  43. 43. rpc.fanout_cast(topic) MatchMaker e.g. "compute" Fanout may have many workers. get_workers as workers Array *topic.worker Direct for each Publisher Topic worker: Consumer PUSH -> PULL Invoker i.e. rpc.cast(topic.worker) Worker (e.g. api) (e.g. nova-compute on compute.host) Fanout Cast. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  44. 44. rpc.fanout_cast(topic) MatchMaker e.g. "compute" Fanout may have many workers. get_workers as workers Array *topic.worker Direct for each Publisher Topic worker: Consumer PUSH -> PULL Invoker i.e. rpc.cast(topic.worker) Worker (e.g. api) (e.g. nova-compute on compute.host) Fanout Cast. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  45. 45. rpc.fanout_cast(topic) MatchMaker e.g. "compute" Fanout may have many workers. get_workers as workers Array *topic.worker Direct for each Publisher Topic worker: Consumer PUSH -> PULL Invoker i.e. rpc.cast(topic.worker) Worker (e.g. api) (e.g. nova-compute on compute.host) Fanout Cast. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  46. 46. rpc.fanout_cast(topic) MatchMaker e.g. "compute" Fanout may have many workers. get_workers as workers Array *topic.worker Direct for each Publisher Topic worker: Consumer PUSH -> PULL Invoker i.e. rpc.cast(topic.worker) Worker (e.g. api) (e.g. nova-compute on compute.host) Fanout Cast. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  47. 47. rpc.fanout_cast(topic) MatchMaker e.g. "compute" Fanout may have many workers. get_workers as workers Array *topic.worker Direct for each Publisher Topic worker: Consumer PUSH -> PULL Invoker i.e. rpc.cast(topic.worker) Worker (e.g. api) (e.g. nova-compute on compute.host) Fanout Cast. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  48. 48. rpc.fanout_cast(topic) MatchMaker e.g. "compute" Fanout may have many workers. get_workers as workers Array *topic.worker Direct for each Publisher Topic worker: Consumer PUSH -> PULL Invoker i.e. rpc.cast(topic.worker) Worker (e.g. api) (e.g. nova-compute on compute.host) Fanout Cast. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  49. 49. http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  50. 50. http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  51. 51. NOVA RPC http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12
  52. 52. Thank You Essex Plug-in Download https://github.com/cloudscaling/nova-mq Slide deck: http://bit.ly/HJogQ8 CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attributionTuesday, April 17, 12

×