Your SlideShare is downloading. ×
0
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
ZeroMQ: Super Sockets - by J2 Labs
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

ZeroMQ: Super Sockets - by J2 Labs

9,216

Published on

0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
9,216
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
259
Comments
0
Likes
19
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. ZeroMQ: Super Sockets<br />By James Dennis (@j2labs)<br />http://j2labs.net<br />
  • 2. What is ZeroMQ?<br />Message Passing<br />You still design the payload<br />Fault tolerant<br />A message IS your payload<br />even when broken into multiple packets<br />Many messaging “patterns” offered<br />
  • 3. What is ZeroMQ?<br />Super Sockets<br />Multiple networking protocols(tcp, ipc, whatevz!)<br />Language agnostic (bindings for: ruby, python, java, c, lua, lisp, scheme and more…)<br />Feels like a scripting language for sockets<br />Socket behaviors<br />Round-robin routing<br />Publishing to subscribers<br />Sending job requests<br />
  • 4. What ZeroMQ is not…<br />Persistent<br />reasonably easy to add<br />A message queue<br />there are literally zero mq’s here<br />An out-of-the-box solution<br />instead, it’s easy to build any solution<br />remember, they’re like scripting sockets<br />
  • 5. Socket Types (transports)<br />tcp :: regular TCP sockets<br />socket.connect(“tcp://…”)<br />ipc :: interprocess communication (unix sockets)<br />socket.connect(“ipc://…”)<br />in-proc :: in process communication<br />socket.connect(“in-proc://…”)<br />pgm – pragmatic multicast (or epgmvia udp)<br />socket.connect(“pgm://…”)<br />
  • 6. Socket Behaviors<br />Actually called patterns<br />They exist to make typical cases easy<br />Request-Reply<br />Publish-Subscribe<br />Pipeline<br />Exclusive pair<br />you probably don’t want this<br />
  • 7. Patterns: PUSH/PULL<br />Sends messages to multiple connected hosts<br />Passes messages ROUND ROBIN to connections<br />Blocks PUSH socket if no connections are present<br />N hosts PULL messages from pusher<br />Connect/Remove new hosts as necessary<br />
  • 8. Patterns: PUSH/PULL<br />Server: Sends a message<br />import zmq<br />ctx = zmq.Context()<br />s = ctx.socket(zmq.PUSH)<br />s.bind("ipc://*:5678")<br />while True:<br />s.send(”blablabla") # blocks if no receivers<br /> print “Sent one”<br />
  • 9. Patterns: PUSH/PULL<br />Client: Prints messages from server socket<br />import zmq<br />ctx = zmq.Context()<br />s = ctx.socket(zmq.PULL)<br />s.connect("ipc://*:5678")<br />while True:<br />msg = s.recv()<br /> print 'Got msg:', msg<br />
  • 10. Patterns: PUSH/PULL<br />A typical worker pipeline <br />looks a bit like this.<br />What’s that PUB/SUB stuff?<br />
  • 11. Patterns: PUB/SUB<br />PUB socket sends topical payloads<br />Can start streaming messages right away<br />ZMQ discards messages if no subscribers exist<br />SUB socket subscribes to “topic”<br />receives all messages<br />reads topic and discards accordingly<br />happens under the hood.<br />
  • 12. Patterns: PUB/ SUB<br />Server: sends messages to `whatevz`<br />s = ctx.socket(zmq.PUB)<br />s.bind("tcp://127.0.0.1:5566")<br />topic = ’whatevz’<br />message = json.dumps({'msg': 'what up dude?'})<br />while True:<br />s.send(topic, message) # send to topic<br /> print ‘Sent one’<br />
  • 13. Patterns: PUB/SUB<br />Client: demonstrates subscribing to `whatevz`<br />s = ctx.socket(zmq.SUB)<br />s.connect("tcp://127.0.0.1:5566")<br />topic = "whatevz"<br />s.setsockopt(zmq.SUBSCRIBE, topic) # set socket option<br />while True:<br />msg = s.recv()<br /> print 'Got msg:’, json.loads(msg)<br />
  • 14. Patterns: REQ/REP<br />One to one messaging<br />Each REQ message must have a REP response<br />Synchronous communication<br />Will error if two REQ’s are sent before one REP<br />
  • 15. Patterns REQ/REP<br />Client<br />context.socket(zmq.REQ) # btw… REQ sockets are blocking!<br />Worker(could say “server”)<br />context.socket(zmq.REP)<br />Multiple Connections<br />One-to-one messaging is useful for workers getting work<br />Basically, pull work request & reply when done<br />Client may need to multiple workers<br />Can’t just open more sockets… we need a new pattern… <br />
  • 16. Patterns: REQ/REP<br />CLIENTsends a “Hello”<br />SERVERsends a “world” back<br />The REQ socket can send a message, but must not send anything else until it gets a response or ZMQ will error.<br />Likewise, a REP socket cannot send a message unless it first received one.<br />
  • 17. Synchronized PUB/SUB<br />Handling “no broker” (aka building one)<br />More than one socket used<br />Like having OneMQ now<br />Hypothetical Communication Order:<br />Subscriber sends sub REQ for topic<br />Publisher REP - “listen <tcp://here>”<br />Subscriber listens to PUB socket<br />
  • 18. Patterns: XREQ/XREP<br />XREQ is a historical name for DEALER<br />A DEALER asynchronously sends to *REP sockets<br />It can send to XREP (aka: ROUTER) sockets too<br />XREP is a historical name for ROUTER<br />A ROUTER asynchronously handles *REQ sockets<br />It can handle DEALER (aka: XREQ) sockets too.<br />
  • 19. Patterns: DEALER/ROUTER<br /><ul><li>CLIENT sends message to ROUTER and waits for a response
  • 20. ROUTER processes message and sends to some worker via the DEALER
  • 21. DEALERsends message to worker and waits for one response
  • 22. WORKER processes message, sends response back
  • 23. DEALER get message from WORKER, sends along to ROUTER
  • 24. ROUTERgets message and sends back to CLIENT
  • 25. CLIENTdoes something with it!</li></li></ul><li>Complicated Network Layouts<br />
  • 26. Broker Model: aka “The Titanic”<br />
  • 27. Broker Model with ZMQ<br />Every connection is a REQ socket<br />ROUTER socket to all connections<br />LRU Queue (your broker code!)<br />Up to the LRU Queue handle socket identities properly<br />REQ/REP gets that for free.<br />
  • 28. Flexibility Via Decentralization<br />“ØMQ is like Git(essentially distributed) whereas AMQP is like SVN (essentially centralized)” – James Dennis (me)<br />Welcome From AMQP: http://www.zeromq.org/docs:welcome-from-amqp<br />
  • 29. Mongrel2 (Zed Shaw’s second web server)<br />http://mongrel2.org<br />
  • 30. Brubeck<br />Mongrel2 handler in Python<br />Modeled after Tornado with ideas from Flask / Bottle<br />Uses Eventlet<br />Coroutines!<br />Nonblocking I/O!<br />Uses ZeroMQ<br />Write back end services in any language!<br />Uses DictShield<br />I wrote this too! <br />Database agnostic modeling! <br />ZeroMQ is language agnostic, right?!<br />https://github.com/j2labs/brubeck<br />
  • 31. Try it!<br />https://github.com/j2labs/zmq_examples<br />http://mongrel2.org<br />https://github.com/j2labs/brubeck<br />Read more!<br />http://nichol.as/zeromq-an-introduction<br />http://zguide.zeromq.org/<br />
  • 32. Questions<br />?<br />James Dennis (@j2labs)<br />jdennis@gmail.com<br />

×