The document discusses distributed RPC in OpenStack Nova using ZeroMQ. It proposes a solution of using decentralized peer-to-peer communication between the Nova components rather than relying on a centralized message broker. This involves using ZeroMQ for direct topic subscription between the Nova Network, Compute, and Scheduler components. It also describes using matchmaking and fanout casting to distribute messages to multiple worker nodes.

1. X
Image: http://ken.pepple.info/openstack/2011/04/22/openstack-nova-architecture/
Distributed RPC in Nova
with ZeroMQ
04-16-2012
Distributed RPC with ZeroMQ: code CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
& analysis 1
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

2. Etherpad
http://etherpad.openstack.org/
FolsomDistributedRPC
04-16-2012
Distributed RPC with ZeroMQ: code CCA - NoDerivs 3.0 Unported License -2Usage OK, no modifications, full attribution
& analysis
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

3. @ewindisch
04-16-2012
Distributed RPC with ZeroMQ: code CCA - NoDerivs 3.0 Unported License -3Usage OK, no modifications, full attribution
& analysis
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

4. Etherpad
http://etherpad.openstack.org/
FolsomDistributedRPC
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

5. Application
Message
Queue
Broker
Brokered Messaging
04-16-2012
Distributed RPC with ZeroMQ: code CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
& analysis 5
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

6. Application
X
Message
Queue
Broker
Brokered Messaging
04-16-2012
Distributed RPC with ZeroMQ: code CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
& analysis 5
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

7. Application
Message
Queue
Brokered Messaging
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

8. Application
Brokered Messaging
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

9. Application
HA Failover
High-Availability
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

10. Application
failover fails
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

11. Nova Nova Nova
API
Network Compute Scheduler
Message Message
Queue Queue
Nova Nova Nova
API
Network Compute Scheduler
Application
Decentralized
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

12. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

13. CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

14. Nova Nova Nova
API
Network Compute Scheduler
Message Message
Queue Queue
Nova Nova Nova
API
Network Compute Scheduler
Application
Decentralized
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

15. Nova Nova Nova
API
Network Compute Scheduler
Message
Abandoned
Queue Messages
Nova Nova Nova
API
Network Compute Scheduler
Application
Decentralized
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

16. Nova Nova Nova
API
Network Compute Scheduler
Message
Queue
Nova Nova Nova
API
Network Compute Scheduler
Application
Decentralized
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

17. Solution:
Distributed RPC
for OpenStack Nova
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

18. API
Nova API
Application
Nova
Nova Network Nova Scheduler
Network
Nova
Nova Compute
Compute
Peer2Peer
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

19. API
Nova API
Application
Nova
Nova Network
Network
Nova
Nova Compute
Compute
Peer2Peer
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

20. API
Nova API
Application
Nova Scheduler
Nova
Nova Network
Network
Nova Scheduler
Nova
Nova Compute
Compute
Peer2Peer
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

21. API
Nova API
Application
Nova
Nova Network
Network
Nova Scheduler
Nova
Nova Compute
Compute
Peer2Peer
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

22. rpc.cast(topic.host)
PUSH -> PULL Direct
Publisher Topic
Consumer
Invoker
Worker
(e.g. api)
(e.g. compute)
Cast to host.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

23. rpc.cast(topic.host)
PUSH -> PULL Direct
Publisher Topic
Consumer
Invoker
Worker
(e.g. api)
(e.g. compute)
Cast to host.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

24. rpc.cast(topic.host)
PUSH -> PULL Direct
Publisher Topic
Consumer
Invoker
Worker
(e.g. api)
(e.g. compute)
Reply.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

25. rpc.cast(topic.host)
PUSH -> PULL Direct
Publisher Topic
Consumer
Invoker
Worker
(e.g. api)
(e.g. compute)
Reply.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

26. def _send(...):
“”” code omitted “””
# Curry the original request into a reply method.
orig_payload = [RpcContext.marshal(context), msg]
payload = [RpcContext.marshal(context), {
'method': '-reply', # Send original call to wrapper
'args': {
'msg_id': msg_id,
'context': RpcContext.marshal(context),
'topic': reply_topic, # our return path
'msg': orig_payload
}
}]
Tuesday, April 17, 12

27. # Messages arriving async.
msg_waiter = QueueSocket(
"ipc:///var/run/nova/zmq_reply_queue",
zmq.SUB, subscribe=msg_id, bind=False)
try:
with Timeout(timeout):
with Timeout(30) as t1:
conn.cast(msg_id, topic, payload)
# Blocks until receives reply
responses = pickle.loads(msg_waiter.recv()[-1])
Tuesday, April 17, 12

28. def reply(...): # this is -reply
"""Reply to a casted call"""
child_ctx = RpcContext.unmarshal(orig_payload[0])
response = ConsumerBase.normalize_reply(
self.get_response( # execute original call
child_ctx, proxy, topic, orig_payload[1]),
ctx.replies
)
_multi_send("cast", ctx, reply_topic, {
'method': '-process_reply',
'args': {
'msg_id': msg_id,
'response': response
}
})
Tuesday, April 17, 12

29. def process_reply(...):
"""Process a reply"""
# zmq_replies is the receiver service
self.msg_waiter.cast(str(msg_id),
str('zmq_replies'), response)
Tuesday, April 17, 12

30. rpc.call(topic.worker)
PUSH->PULL
curries reply method
Direct
Publisher Topic
Consumer
PULL<-PUSH
Invoker
rpc.cast -process_reply() Worker
(e.g. api)
(e.g. compute.host)
Call method.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

31. rpc.call(topic.worker)
PUSH->PULL
curries reply method
Direct
Publisher Topic
Consumer
PULL<-PUSH
Invoker
rpc.cast -process_reply() Worker
(e.g. api)
(e.g. compute.host)
Call method.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

32. rpc.call(topic.worker)
PUSH->PULL
curries reply method
Direct
Publisher Topic
Consumer
PULL<-PUSH
Invoker
rpc.cast -process_reply() Worker
(e.g. api)
(e.g. compute.host)
Call method.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

33. rpc.call(topic.worker)
PUSH->PULL
curries reply method
Direct
Publisher Topic
Consumer
PULL<-PUSH
Invoker
rpc.cast -process_reply() Worker
(e.g. api)
(e.g. compute.host)
Call method.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

34. Bare Topics
“scheduler.host1” goes to “host1”
Where do messages for “scheduler” go?
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

35. MatchMaking
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

36. Rewrite
Engine
• Every incoming topic is
mapped to an array of topics
and hosts.
• Usually returns a single topic/
host.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

37. Search for hosts
• Can be provided by:
• hash table (implemented)
• name service (i.e. DNS)
• directory service (i.e. LDAP)
• zookeeper
• etc...
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

38. rpc.fanout_cast(topic) MatchMaker
e.g. "compute"
Fanout may have many workers.
get_workers
as workers
Array *topic.worker
Direct
for each
Publisher Topic
worker:
Consumer
PUSH -> PULL
Invoker i.e. rpc.cast(topic.worker)
Worker
(e.g. api)
(e.g. nova-compute on
compute.host)
Fanout Cast.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

39. rpc.fanout_cast(topic) MatchMaker
e.g. "compute"
Fanout may have many workers.
get_workers
as workers
Array *topic.worker
Direct
for each
Publisher Topic
worker:
Consumer
PUSH -> PULL
Invoker i.e. rpc.cast(topic.worker)
Worker
(e.g. api)
(e.g. nova-compute on
compute.host)
Fanout Cast.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

40. rpc.fanout_cast(topic) MatchMaker
e.g. "compute"
Fanout may have many workers.
get_workers
as workers
Array *topic.worker
Direct
for each
Publisher Topic
worker:
Consumer
PUSH -> PULL
Invoker i.e. rpc.cast(topic.worker)
Worker
(e.g. api)
(e.g. nova-compute on
compute.host)
Fanout Cast.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

41. rpc.fanout_cast(topic) MatchMaker
e.g. "compute"
Fanout may have many workers.
get_workers
as workers
Array *topic.worker
Direct
for each
Publisher Topic
worker:
Consumer
PUSH -> PULL
Invoker i.e. rpc.cast(topic.worker)
Worker
(e.g. api)
(e.g. nova-compute on
compute.host)
Fanout Cast.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

42. rpc.fanout_cast(topic) MatchMaker
e.g. "compute"
Fanout may have many workers.
get_workers
as workers
Array *topic.worker
Direct
for each
Publisher Topic
worker:
Consumer
PUSH -> PULL
Invoker i.e. rpc.cast(topic.worker)
Worker
(e.g. api)
(e.g. nova-compute on
compute.host)
Fanout Cast.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

43. rpc.fanout_cast(topic) MatchMaker
e.g. "compute"
Fanout may have many workers.
get_workers
as workers
Array *topic.worker
Direct
for each
Publisher Topic
worker:
Consumer
PUSH -> PULL
Invoker i.e. rpc.cast(topic.worker)
Worker
(e.g. api)
(e.g. nova-compute on
compute.host)
Fanout Cast.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

44. rpc.fanout_cast(topic) MatchMaker
e.g. "compute"
Fanout may have many workers.
get_workers
as workers
Array *topic.worker
Direct
for each
Publisher Topic
worker:
Consumer
PUSH -> PULL
Invoker i.e. rpc.cast(topic.worker)
Worker
(e.g. api)
(e.g. nova-compute on
compute.host)
Fanout Cast.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

45. rpc.fanout_cast(topic) MatchMaker
e.g. "compute"
Fanout may have many workers.
get_workers
as workers
Array *topic.worker
Direct
for each
Publisher Topic
worker:
Consumer
PUSH -> PULL
Invoker i.e. rpc.cast(topic.worker)
Worker
(e.g. api)
(e.g. nova-compute on
compute.host)
Fanout Cast.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

46. rpc.fanout_cast(topic) MatchMaker
e.g. "compute"
Fanout may have many workers.
get_workers
as workers
Array *topic.worker
Direct
for each
Publisher Topic
worker:
Consumer
PUSH -> PULL
Invoker i.e. rpc.cast(topic.worker)
Worker
(e.g. api)
(e.g. nova-compute on
compute.host)
Fanout Cast.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

47. rpc.fanout_cast(topic) MatchMaker
e.g. "compute"
Fanout may have many workers.
get_workers
as workers
Array *topic.worker
Direct
for each
Publisher Topic
worker:
Consumer
PUSH -> PULL
Invoker i.e. rpc.cast(topic.worker)
Worker
(e.g. api)
(e.g. nova-compute on
compute.host)
Fanout Cast.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

48. rpc.fanout_cast(topic) MatchMaker
e.g. "compute"
Fanout may have many workers.
get_workers
as workers
Array *topic.worker
Direct
for each
Publisher Topic
worker:
Consumer
PUSH -> PULL
Invoker i.e. rpc.cast(topic.worker)
Worker
(e.g. api)
(e.g. nova-compute on
compute.host)
Fanout Cast.
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

49. http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html
http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

50. http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html
http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

51. NOVA
RPC
http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html
http://amitpiplani.blogspot.com/2010/05/u-pick-2-selection-for-nosql-providers.html
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12

52. Thank You
Essex Plug-in Download
https://github.com/cloudscaling/nova-mq
Slide deck:
http://bit.ly/HJogQ8
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution
Tuesday, April 17, 12