ZeroMQ is a library for message passing between processes using sockets. It provides patterns for common messaging needs like publish-subscribe, request-reply, and push-pull. ZeroMQ supports multiple protocols including TCP and inter-process communication. It aims to be more flexible than traditional messaging systems by allowing decentralized architectures. Examples of projects using ZeroMQ include Mongrel2, a high-performance web server, and Brubeck, a Python web framework.

1. ZeroMQ: Super Sockets By James Dennis (@j2labs) http://j2labs.net

2. What is ZeroMQ? Message Passing You still design the payload Fault tolerant A message IS your payload even when broken into multiple packets Many messaging “patterns” offered

3. What is ZeroMQ? Super Sockets Multiple networking protocols(tcp, ipc, whatevz!) Language agnostic (bindings for: ruby, python, java, c, lua, lisp, scheme and more…) Feels like a scripting language for sockets Socket behaviors Round-robin routing Publishing to subscribers Sending job requests

4. What ZeroMQ is not… Persistent reasonably easy to add A message queue there are literally zero mq’s here An out-of-the-box solution instead, it’s easy to build any solution remember, they’re like scripting sockets

5. Socket Types (transports) tcp :: regular TCP sockets socket.connect(“tcp://…”) ipc :: interprocess communication (unix sockets) socket.connect(“ipc://…”) in-proc :: in process communication socket.connect(“in-proc://…”) pgm – pragmatic multicast (or epgmvia udp) socket.connect(“pgm://…”)

6. Socket Behaviors Actually called patterns They exist to make typical cases easy Request-Reply Publish-Subscribe Pipeline Exclusive pair you probably don’t want this

7. Patterns: PUSH/PULL Sends messages to multiple connected hosts Passes messages ROUND ROBIN to connections Blocks PUSH socket if no connections are present N hosts PULL messages from pusher Connect/Remove new hosts as necessary

8. Patterns: PUSH/PULL Server: Sends a message import zmq ctx = zmq.Context() s = ctx.socket(zmq.PUSH) s.bind("ipc://*:5678") while True: s.send(”blablabla") # blocks if no receivers print “Sent one”

9. Patterns: PUSH/PULL Client: Prints messages from server socket import zmq ctx = zmq.Context() s = ctx.socket(zmq.PULL) s.connect("ipc://*:5678") while True: msg = s.recv() print 'Got msg:', msg

10. Patterns: PUSH/PULL A typical worker pipeline looks a bit like this. What’s that PUB/SUB stuff?

11. Patterns: PUB/SUB PUB socket sends topical payloads Can start streaming messages right away ZMQ discards messages if no subscribers exist SUB socket subscribes to “topic” receives all messages reads topic and discards accordingly happens under the hood.

12. Patterns: PUB/ SUB Server: sends messages to `whatevz` s = ctx.socket(zmq.PUB) s.bind("tcp://127.0.0.1:5566") topic = ’whatevz’ message = json.dumps({'msg': 'what up dude?'}) while True: s.send(topic, message) # send to topic print ‘Sent one’

13. Patterns: PUB/SUB Client: demonstrates subscribing to `whatevz` s = ctx.socket(zmq.SUB) s.connect("tcp://127.0.0.1:5566") topic = "whatevz" s.setsockopt(zmq.SUBSCRIBE, topic) # set socket option while True: msg = s.recv() print 'Got msg:’, json.loads(msg)

14. Patterns: REQ/REP One to one messaging Each REQ message must have a REP response Synchronous communication Will error if two REQ’s are sent before one REP

15. Patterns REQ/REP Client context.socket(zmq.REQ) # btw… REQ sockets are blocking! Worker(could say “server”) context.socket(zmq.REP) Multiple Connections One-to-one messaging is useful for workers getting work Basically, pull work request & reply when done Client may need to multiple workers Can’t just open more sockets… we need a new pattern…

16. Patterns: REQ/REP CLIENTsends a “Hello” SERVERsends a “world” back The REQ socket can send a message, but must not send anything else until it gets a response or ZMQ will error. Likewise, a REP socket cannot send a message unless it first received one.

17. Synchronized PUB/SUB Handling “no broker” (aka building one) More than one socket used Like having OneMQ now Hypothetical Communication Order: Subscriber sends sub REQ for topic Publisher REP - “listen <tcp://here>” Subscriber listens to PUB socket

18. Patterns: XREQ/XREP XREQ is a historical name for DEALER A DEALER asynchronously sends to *REP sockets It can send to XREP (aka: ROUTER) sockets too XREP is a historical name for ROUTER A ROUTER asynchronously handles *REQ sockets It can handle DEALER (aka: XREQ) sockets too.

20. ROUTER processes message and sends to some worker via the DEALER

21. DEALERsends message to worker and waits for one response

22. WORKER processes message, sends response back

23. DEALER get message from WORKER, sends along to ROUTER

24. ROUTERgets message and sends back to CLIENT

26. Broker Model: aka “The Titanic”

27. Broker Model with ZMQ Every connection is a REQ socket ROUTER socket to all connections LRU Queue (your broker code!) Up to the LRU Queue handle socket identities properly REQ/REP gets that for free.

28. Flexibility Via Decentralization “ØMQ is like Git(essentially distributed) whereas AMQP is like SVN (essentially centralized)” – James Dennis (me) Welcome From AMQP: http://www.zeromq.org/docs:welcome-from-amqp

29. Mongrel2 (Zed Shaw’s second web server) http://mongrel2.org

30. Brubeck Mongrel2 handler in Python Modeled after Tornado with ideas from Flask / Bottle Uses Eventlet Coroutines! Nonblocking I/O! Uses ZeroMQ Write back end services in any language! Uses DictShield I wrote this too! Database agnostic modeling! ZeroMQ is language agnostic, right?! https://github.com/j2labs/brubeck

31. Try it! https://github.com/j2labs/zmq_examples http://mongrel2.org https://github.com/j2labs/brubeck Read more! http://nichol.as/zeromq-an-introduction http://zguide.zeromq.org/

32. Questions ? James Dennis (@j2labs) jdennis@gmail.com