This document discusses the current state and further development of the ZMQ driver used by Mirantis. It describes changes made to reduce dependencies on eventlet and use a more flexible internal architecture based on ZeroMQ socket patterns. The document outlines the driver's new message processing path and discusses keeping a proxy component to avoid blocking. It also describes the driver's internal architecture including publisher-consumer pairs for different messaging patterns. Areas needing work for production readiness are identified as heartbeats, retries, persistence, performance optimization, and testing HA and scaling.

1. Copyright © 2015 Mirantis, Inc. All rights reserved
www.mirantis.com
ZMQ driver
Current state and further development
Oleksii Zamiatin
Software Engineer
ozamiatin@mirantis.com
IRC, launchpad: ozamiatin

2. Copyright © 2015 Mirantis, Inc. All rights reserved
Well known HLD

3. Copyright © 2015 Mirantis, Inc. All rights reserved
Old Driver Message Path
request
reply

4. Copyright © 2015 Mirantis, Inc. All rights reserved
New Driver Message Path
request
reply

5. Copyright © 2015 Mirantis, Inc. All rights reserved
What Have Changed
Reduced Proxy-receiver. Dynamic port binding instead of
static + proxy. (Thanks Alec Hothan for the idea! :))
Changed PUSH/PULL+PUB/SUB to REQ/REP BP:zmq-req-
rep
Internal architecture become more flexible - easy to
extend BP:zmq-patterns-usage, BP:zmq-driver-folder
Not depends on eventlet directly BP:zmq-work-without-eventlet
More clear topics management and message serialization

6. Copyright © 2015 Mirantis, Inc. All rights reserved
Client proxy to avoid blocking
request
reply
cfg.BoolOpt('zmq_use_broker', default=True) - controlled by the option

7. Copyright © 2015 Mirantis, Inc. All rights reserved
Why we still need a proxy?
Eventlet
libzmq - c-library
BLOCKING call to DEALER by specification
DEALER socket blocks when no listeners present
eventlet.green.zmq - wouldn’t handle the situation…

8. Copyright © 2015 Mirantis, Inc. All rights reserved
Driver’s architecture from the inside
So-called pipeline. Driver consists of publisher-consumer pairs,
each implements some OM patterns in terms of ZMQ.

9. Copyright © 2015 Mirantis, Inc. All rights reserved
Driver’s architecture from the inside
Basic publishers/consumers compatibility
Based on ZeroMQ socket types compatibility

10. Copyright © 2015 Mirantis, Inc. All rights reserved
Driver’s architecture from the inside
Each pipeline serves for some subset of OM patterns. Overlapping takes place.
Based on ZeroMQ socket types compatibility
Patterns:
R - CALL
C - CAST
Cf - CAST+Fanout
N - Notify
Nf - Notify+Fanout
<-,C,Cf,N,Nf>
<R, - , - , - , - >
< - , - ,Cf, - ,Nf>
< - ,C, - ,N, - >

11. Copyright © 2015 Mirantis, Inc. All rights reserved
Driver’s architecture from the inside
How can the old driver’s pattern fit into current architecture
Patterns:
R - CALL
C - CAST
Cf - CAST+Fanout
N - Notify
Nf - Notify+Fanout
<R, C, Cf, N, Nf>
Hybrid publishers/consumers are possible with current approach

12. Copyright © 2015 Mirantis, Inc. All rights reserved
Driver’s architecture from the inside
ZmqClient combines publishers in order to make a complete <R,C,Cf,N,Nf>.
Patterns:
R - CALL
C - CAST
Cf - CAST+Fanout
N - Notify
Nf - Notify+Fanout
<-,C,Cf,N,Nf>
<R, - , - , - , - >
< - , - ,Cf, - ,Nf>
< - ,C, - ,N, - >
ZmqServer should match the client with corresponding consumers combination.

13. Copyright © 2015 Mirantis, Inc. All rights reserved
Driver’s architecture from the inside
Patterns:
R - CALL
C - CAST
Cf - CAST+Fanout
N - Notify
Nf - Notify+Fanout
<-,C,Cf,N,Nf>
<R, - , - , - , - >
< - , - ,Cf, - ,Nf>
< - ,C, - ,N, - >
Just as an example

14. Copyright © 2015 Mirantis, Inc. All rights reserved
Driver’s architecture Client

15. Copyright © 2015 Mirantis, Inc. All rights reserved
Driver’s architecture Server

16. Copyright © 2015 Mirantis, Inc. All rights reserved
Driver’s architecture from the inside
What we have now - the basic “reliable” pipeline to start from
More reliable but
slower
Unreliable but faster
Blocking, so makes sense
only with blocking executor
<-,C,Cf,N,Nf>
<R, - , - , - , - >

17. Copyright © 2015 Mirantis, Inc. All rights reserved
What ZMQ doesn’t cover
Comparatively to RabbitMQ, ZMQ doesn’t provide:
Nodes discovery and management
Users management and security (just experimental
encryption functionality)
Persistence of messaging queues
Supporting of this functionality makes zmq-driver more
complicated.

18. Copyright © 2015 Mirantis, Inc. All rights reserved
What should be done for production-readiness
Heartbeats, keeping connections, reconnects
Acks based retries (in progress)
Persistence for queues (at least research possibilities)
Performance optimizations (optimize calls to Redis
implement ttl-based cache)
Testing HA, scaling.
Fix all bugs in the list