This document discusses messaging patterns using ZeroMQ in Python. It begins with background on messaging and common patterns. ZeroMQ features lightweight overhead, auto-reconnect, different transports including TCP/IP, IPC, and inter-thread. Common patterns with ZeroMQ include REQ/REP for request-response, SUB/PUB for publish-subscribe, PUSH/PULL for routing, and PAIR for bidirectional communication. Examples demonstrate using different transports and how ZeroMQ supports flexible, high performance, and multi-language messaging architectures.

1. Messaging Patterns with ZeroMQ
In Python
“It’s time to think distributedly”
By Victor Lin

2. The Age of Distribution

3. Different messaging methods
Node Node
Process Process
TCP/IP
UDP
IPC
Network
UDP
Process
Node Node Node
Thread Thread
ITC

4. Background
• It’s all about messaging
• Too many messaging transports
• Common patterns cannot be reused
• Auto reconnect is needed

5. Superhero

6. ZeroMQ features
• Lightweight overhead
• Auto reconnect
• One-to-many connection
• Support different transports
• Support different patterns
• Support different languages

7. Patterns
• REQ/REP
• SUB/PUB
• PUSH/PULL
• PAIR

8. REQ/REP
• Round-robin
• One-to-many
• One request, one response

9. SUB/PUB
• One-to-many
• All subscribers receive messages from
publisher

10. PUSH/PULL
• Round-robin
• One-to-many
• Many-to-one

11. PAIR
• Just like BSD socket
• Bidirectional communication

12. Different transports
• TCP/IP – tcp://
• Unix domain socket (IPC) – ipc://
• Inter-thread Communication – inproc://
• Multicast – pgm://

13. Same program, different transport
Node Node
Process Process
tcp://
ipc://
pgm://
Network
Process
Node Node Node
Thread Thread
inproc://

14. TCP/IP
Worker
tcp:// Node
Manager Worker
Node tcp:// REQ/REP tcp:// Node
Worker
tcp:// Node

15. Unix domain socket
Node
Worker
ipc:// Process
Manager Worker
Process ipc:// REQ/REP ipc:// Process
Worker
ipc:// Process

16. Inter-thread Communication
Node
Process
Worker
inproc:// Thread
Manager Worker
Thread inproc:// REQ/REP inproc:// Thread
Worker
inproc:// Thread

17. Different languages
• Python
• C/C++
• Java
• .Net
• Ruby
• Go
• PHP
• Perl
• …

18. Example: authentication service
• Updating authentication module requires
restarting server
• Authentication module could be the
bottleneck
Broadcast
Mr.DJ TCP/IP Authentication
module

19. Example: authentication service
• Authentication service can be restarted
anytime
• Working load can be distributed
Authentication
Broadcast REQ/REP (IPC) service
Mr.DJ

20. Example: authentication service
Broadcast Authentication
service
Authentication
Broadcast broker Authentication
service
Broadcast Authentication
service

21. Broker in ZMQ

22. Example: recording service
• Record audio when broadcasting
• Upload file chunks to Amazon S3
• Must be reliable

23. Amazon S3
Broadcast
PUB/SUB (IPC)
Uploading Uploading Uploading
worker worker worker
Writing service
REQ/REP (IPC)
File Uploading
service
Local
DB

24. Server level plugins
• Attach whatever you want
Record service
ipc://
Broadcast ipc:// Audio
PUB/SUB ipc:// identifying
service
ipc://
…

25. Conclusions
• Flexible
• High performance
• Easy to use
• Multi-language supported
• Focus on messaging and topology

26. Q&A
• Any question?