• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
ZeroMQ with NodeJS
 

ZeroMQ with NodeJS

on

  • 15,487 views

 

Statistics

Views

Total Views
15,487
Views on SlideShare
9,477
Embed Views
6,010

Actions

Likes
44
Downloads
264
Comments
2

16 Embeds 6,010

http://css.dzone.com 5458
http://www.eduardoshanahan.com 223
https://twitter.com 180
http://java.dzone.com 108
http://architects.dzone.com 12
http://www.newsblur.com 11
http://rritw.com 5
http://www.dzone.com 3
http://elcurator.octo.com 2
http://webcache.googleusercontent.com 2
http://www.google.cl 1
http://www.rritw.com 1
https://www.google.com 1
https://www.google.de 1
http://cloud.feedly.com 1
http://www.google.ca 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

12 of 2 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    ZeroMQ with NodeJS ZeroMQ with NodeJS Presentation Transcript

    • ØMQ @ NodeJSØMQ: Small Pieces for Scalable SoftwareFernando D. Alonso
    • 1. Facts about ØMQ[ ]
    • Facts about ØMQLow level messaging library○ It is written in C++.○ A thin layer between application and transportlayers.○ Message-passing using in-memory queues.○ Concise API which talks over the ZMTP protocol.○ Works on top of TCP, IPC, in-memory, andPGM/EPGM communication protocols.
    • ○ Philosophy: horizontal scalability to reduce pointsof failure.○ Network topology aware. Point to pointcommunication reduce latencies.○ Communication is reduced to a few patterns, whichcombined are powerful.○ No need to have a multitenant service.Facts about ØMQA broker is not mandatory
    • Facts about ØMQMessaged oriented communication○ Messages as first-class citizen. No need to fightwith framing or buffering.○ You can send multipart messages.○ Messages are atomic.○ Asynchronous messaging. A single background I/Othread does the job for ØMQ sockets."A ØMQ message is a discrete unit of data passed betweenapplications or components of the same application. ØMQ messageshave no internal structure and from the point of view of ØMQ itselfthey are considered to be opaque binary data."
    • Facts about ØMQFast for development○ Interchangeable transports: e.g, scaling from oneIPC server to a bunch of TCP.○ Automatic reconnections.○ Multicore made easy.○ Provides zmq_poll to read BSD sockets.
    • ○ It doesnt provide message persistence.○ It doesnt provide data serialization.○ It doesnt provide data compression.○ It doesnt provide message encryption.○ It doesnt provide security mechanisms. Not in thenext version of the ZMTP protocol !(*) ZTMP 3.0 draft protocol: http://hintjens.com/blog:39Facts about ØMQAims for simplicity
    • Facts about ØMQPlatforms & Languages○ Bindings for 30+ languages including NodeJS, Ruby,Python, Java, C, C++, Scala, Erlang, Perl, PHP, .NET.○ Linux & Windows.○ Raspberry PI.○ Android OS.
    • 2. Install[ , ]
    • InstallOn Linux from sourcesudo apt-get install build-essential libtool autoconf automake uuid-devwget http://download.zeromq.org/zeromq-3.2.3.tar.gztar xvzf zeromq-3.2.3.tar.gz./configuremakesudo make installsudo ldconfigNOTES:(*) Check http://download.zeromq.org/ for more versions.(**) Avoid 3.0.x, 3.1.x, 3.2.0, 3.2.1 versions. They lack backwardscompatibility.(***) Check ./configure --help for more options on build.(****) ./configure --with-pgm
    • InstallOn Linux using packagesLibzmq 3.2:● Debian Sid: sudo apt-get install libzmq3-devLibzmq 2.2:● Debian Wheezy, Jessie, Sid: sudo apt-get install libzmq-dev● Debian Squeeze backport: http://packages.debian.org/squeeze-backports/libzmq1
    • InstallOn Macbrew install zmq
    • InstallNodeJS bindingsNPM Packages:● zmq (https://github.com/JustinTulloss/zeromq.node) [RECOMMENDED]● zmq-stream (https://github.com/Schoonology/zmq-stream)npm install zmqnode> var zmq = require(zmq);
    • InstallBindings in other languagesRuby: https://github.com/chuckremes/ffi-rzmq [RECOMMENDED]Python: https://github.com/zeromq/pyzmqErlang: https://github.com/zeromq/erlzmq2Java JZMQ (uses JNI): https://github.com/zeromq/jzmqJava JeroMQ: https://github.com/zeromq/jeromqScala ZeroMQ: https://github.com/mDialog/scala-zeromqScala ZeroMQ (official): https://github.com/valotrading/zeromq-scala-binding...
    • 3. Sockets
    • SocketsCreation(*) http://api.zeromq.org/3-2:zmq-socketØMQ C APINodeJS ZMQreq ZMQ_REQrep ZMQ_REPdealer ZMQ_DEALERrouter ZMQ_ROUTERpush ZMQ_PUSHpull ZMQ_PULLØMQ C APINodeJS ZMQpub ZMQ_PUBsub ZMQ_SUBxsub ZMQ_XSUBxpub ZMQ_XPUBvar sock = zmq.socket("<SOCKET_TYPE>");zmq_socket(context, <SOCKET_TYPE>);RequestReplyDealer (async Request)Router (async Reply)PushPullPublisherSubscriberXSubscriberXPublisher
    • SocketsBind & Connectsock.bind("tcp://10.0.0.1:5555",function(err) {..});sock.connect("tcp://10.0.0.1:5555");sock.close();(*) http://api.zeromq.org/3-2:zmq-bind(**) http://api.zeromq.org/3-2:zmq-connect(***) http://api.zeromq.org/3-2:zmq-closeØMQ C APIzmq_bind(socket, "tcp://10.0.0.1:5555");zmq_connect(socket, "tcp://10.0.0.1:5555");zmq_close(socket);NodeJS ZMQBindConnectClose
    • SocketsBind & Connectsock.bindSync("tcp://10.0.0.1:5555");// This line gets executed once the socket// is binded, but it breaks on error.NodeJS ZMQBindSync
    • SocketsTransportsTCP● usage: tcp://<address>:<port>sock.bind("tcp://192.168.1.100:5555");sock.bind("tcp://eth0:5555");// binds to the first eth0 interfacesock.bind("tcp://*:5555");// binds to all interfaces(*) http://api.zeromq.org/3-2:zmq-tcpØMQ C APIzmq_bind(socket, "tcp://192.168.1.100:5555");zmq_bind(socket, "tcp://eth0:5555");zmq_bind(socket, "tcp://*:5555");NodeJS ZMQ
    • SocketsTransportsInter-Process Communication (IPC)● usage: ipc://<address>● Requires RW permissions on the specified path.(*) http://api.zeromq.org/3-2:zmq-ipcsock.bind("ipc:///tmp/mysocket");ØMQ C APIzmq_bind(socket, "ipc:///tmp/mysocket");NodeJS ZMQ
    • SocketsTransportsIn-process Communication (in-memory)● usage: inproc://<address>● Requires to bind before connect.● Max. address name length is 256.(*) http://api.zeromq.org/3-2:zmq-inproc(**) Buggy on node-zmq package.sock.bind("inproc://queue");ØMQ C APIzmq_bind(socket, "ipc://queue");NodeJS ZMQ
    • SocketsTransportsPGM (multicast)● usage: pgm://<address>;<multicast_address>:<port>● Requires sudo privileges to access raw IP sockets.● Needs ØMQ built with PGM extension (./configure --with-pgm).(*) http://api.zeromq.org/3-2:zmq-pgmsock.bind("pgm://192.168.1.100;239.192.1.1:3055");ØMQ C APIzmq_bind(socket, "pgm://192.168.1.100;239.192.1.1:3055");NodeJS ZMQ
    • SocketsTransportsEncapsulated PGM (multicast)● usage: epgm://<address>;<multicast_address>:<port>● Needs ØMQ built with PGM extension (./configure --with-pgm).(*) http://api.zeromq.org/3-2:zmq-pgmsock.bind("epgm://192.168.1.100;239.192.1.1:3055");ØMQ C APIzmq_bind(socket, "epgm://192.168.1.100;239.192.1.1:3055");NodeJS ZMQ
    • SocketsSend & ReceiveSendsock.send("My message");sock.on("message", function(msg) {console.log(msg.toString());});(*) http://api.zeromq.org/3-2:zmq-send(**) http://api.zeromq.org/3-2:zmq-recvReceivesock.on("message", function(msg) {console.log("Received " + msg);});// <msg> is a Buffer object.
    • SocketsMultipart messagesØMQ Queuezmq_send(socket, "Como", 4, ZMQ_SNDMORE);Messages are atomicA ØMQ message is composed of 1 or moremessage parts.zmq_send(socket, "andas", 5, 0);4 ComoPart 15 andasPart 2ØMQ ensures that peers receive either all message parts of a message ornone at all.The total number of message parts is unlimited except by available memory.
    • SocketsMultipart messagessock.send("First part", zmq.ZMQ_SNDMORE);sock.send("Second part");sock.send(["First part", "Second part"]);Sendvar r = sock.send("Como", zmq.ZMQ_SNDMORE);console.log(r);Check the outgoing buffer and socket state:running thiswill output ->> { type: req,_zmq: { state: 0, onReady:[Function] },_outgoing: [ [ <Buffer 43 6f6d 6f>, 2 ] ],_shouldFlush: true,_events: { message: [Function]} }
    • SocketsMultipart messagessock.on("message", function(first_part, second_part) {console.log(first_part.toString());console.log(second_part.toString());});sock.on("message", function() {for(var key in arguments) {console.log("Part " + key + ": " + arguments[key]);};});Receivesock.on("message", function() {var messages = Array.prototype.slice.call(arguments);messages.forEach(function(msg) {console.log("Part:" + msg);});});
    • SocketsBatched messagingØMQ Queue3 Msg"ØMQ batches messages in opportunistic manner. It sends all themessages available at the moment in one go. Latency of subsequentmessages will be improved because sending single batch to the cardis faster then sending lot of small messages."sock.send("Part A",zmq.ZMQ_SNDMORE);sock.send("Msg");sock.send("PartB");6 Part A6 Part BØMQ Socketsendsendsend<write busy><write ready>Client Thread ØMQ I/O Threadsendbatch<write busy>
    • PatternsPlugging multiple transportsExamplevar zmq = require(zmq),pub = zmq.socket(pub);pub.bindSync(tcp://127.0.0.1:5555);pub.bindSync(ipc:///tmp/zmq.sock);setInterval(function() {pub.send("I am polyglot!");}, 1000);sub = zmq.socket(sub);sub.connect(ipc:///tmp/zmq.sock);sub.subscribe();sub.on(message, function(msg) {console.log("Received: " + msg);});var zmq = require(zmq),sub = zmq.socket(sub);sub.connect(tcp://127.0.0.1:5555);sub.subscribe();sub.on(message, function(msg) {console.log(Received: + msg);});
    • SocketsSetting Socket Options(*) http://api.zeromq.org/3-2:zmq-setsockoptSetsockopt● usage (NodeJS ZMQ API): sock.setsockopt(<option>, <value>);or sock.<option> = <value>;sock.identity = monitor-1;sock.setsockopt(identity, monitor-1);Identity:// value can be any string up to 255 length.// identity is required on sockets// connecting to a Router.
    • SocketsSetting Socket OptionsSetsockoptsubscriber.subscribe(MUSIC);subscriber.setsockopt(subscribe,MUSIC);Subscribe:// sets a message filter for subscriber sockets.subscriber.unsubscribe(MUSIC);susbscriber.setsockopt(subscribe,POP);Unsubscribe:// sets off a message filter for subscribersockets.
    • 4. Patterns[ ]
    • PatternsRequest / ReplySynchronous task distributionØ REP #1Ø REP # 2Ø REQ< round robin >...< synchronous >Used when each message needs to be matched with a response. Handlesonly one message at time. Strict send-recv cycle.
    • PatternsRequest / ReplyØ REP #1 Ø REP # 2Ø REQsendrecv (wait response)sendrecv (dont wait response)recv (dont wait response)
    • PatternsRequest / ReplyØ REP #1 Ø REP # 2Ø REQsendrecvrecvsendsend-> REP #1 is downFAILUREREQ #1 will not recover !
    • PatternsRequest / ReplyBasic Request / Replyvar zmq = require(zmq);reply = zmq.socket(rep);reply.bind(tcp://127.0.0.1:5555,function(err) {if (err) throw err;});reply.on(message, function(msg) {console.log(Received: + msg);reply.send(Pong!);});var zmq = require(zmq);request = zmq.socket(req);request.connect(tcp://127.0.0.1:5555);request.send(Ping);request.on(message, function(msg) {console.log(Response: + msg);});
    • PatternsRequest / ReplyDealer socket message deliveryMessages have to be multipart, consisting on: an empty delimiter header,followed by the content body parts.sock.send("Body");Outgoing Queue06 Bodysock.send(new Buffer([]), zmq.ZMQ_SNDMORE);sendsend
    • PatternsRequest / ReplyDealer socket message deliveryOutgoing messages are round-robined among all connected peers.However, sending depends on the availability of the receiver.sock.send(["", "Msg"]);Outgoing Queue3 Msg6 Part APeer Socket A Peer Socket Bsock.send(["", "Bla");sock.send(["", "Msg2"]);sendsendsendTo ATo BTo Asend<writeready><writebusy><writeready>004 Msg20
    • PatternsRequest / ReplyDealer socket message deliveryReconnections are handled automatically: messages are asynchronouslydelivered. Response time does not block sending more messages.Outgoing Queue Peer Socket A Peer Socket Bsend-> Up againsock.send(["", "Msg"]);3 Msgsock.send(["", "Msg2"]);sendsendTo ATo A04 Msg20
    • PatternsRequest / ReplyØ REP #1 Ø REP # 2Ø DEALERenqueue < Msg A >send < Msg B >-> REP #1 is downenqueue < Msg C >send < Msg A, Msg C >send < Msg D >Dealer socket handles peer reconnection automatically. It will sendmessages that queued for that peer once it has established connection.
    • PatternsRequest / Reply> Response from: 5001> Response from: 5001> Response from: 5002> Response from: 5001> Response from: 5001> Response from: 5002> ...Dealer examplevar dealer = zmq.socket(dealer);dealer.connect(tcp://127.0.0.1:5001);dealer.connect(tcp://127.0.0.1:5002);setInterval(function() {dealer.send(["", "Msg"]);}, 1000);dealer.on(message, function(h, msg) {console.log(Response from: + msg);});running thismight output->
    • PatternsRequest / ReplyRouter socket messagingMessages have to be multipart, consisting on: an identity frame, an emptydelimiter frame, followed by the content body parts. This applies to bothincoming and outgoing messages.sock.send("Body");Outgoing Queue06 Bodysock.send("", zmq.ZMQ_SNDMORE);sendsendsock.send("worker-1", zmq.ZMQ_SNDMORE); 8 worker-1send
    • PatternsRequest / ReplyRouter is asynchronousIncoming messages are fair-queued among all connected and availablepeers.Peer Socket A Peer Socket BØMQ RouterIncoming QueueSocketsendsendsend
    • PatternsRequest / ReplyRouter is asynchronousReceiving depends on the availability of the sender.Peer Socket A Peer Socket BØMQ RouterIncoming QueueSocketsendsend
    • PatternsPush / PullUnidirectional data distributionThe Push / Pull pattern fits well on pipelined communication, i.e., when noresponse is required for a given message.ØØ...ØØ
    • PatternsPush / PullBinding the Push socketOutgoing messages are round-robined among all connected and availablepeers.push.send("Msg");push.send("PartB");Outgoing Queue Peer Socket A Peer Socket Bpush.send("Part A",zmq.ZMQ_SNDMORE);push.send("Msg2");3 Msg6 Part A6 Part B4 Msg2sendsendsendsend
    • PatternsPush / PullBinding the Push socketOutgoing messages are round-robined among all connected and availablepeers.Outgoing Queue Peer Socket A Peer Socket B-> Down3 Msg6 Part A6 Part B4 Msg2sendsendsendpush.send("Msg");push.send("PartB");push.send("Part A",zmq.ZMQ_SNDMORE);push.send("Msg2");sendsendsendsend
    • PatternsPush / PullBinding the Push socketDisconnections and reconnections are handled automatically.Outgoing Queue Peer Socket A Peer Socket B4 More6 Part 16 Part 25 Othersendsendsendpush.send("More"); sendpush.send("Part2");push.send("Part 1",zmq.ZMQ_SNDMORE);sendsendpush.send("Other");send
    • PatternsPush / PullParallel task distributionØ PULL #1Ø PULL # 2Ø PUSH< round robin >...
    • PatternsPush / PullPassive agents / MonitorØ PULLØ PUSH...Ø PUSH
    • PatternsPush / PullBinding the Pull socketPushs outgoing messages are round-robined among all connected peers.push.send("Msg");push.send("PartB");Outgoing Queue Peer Socket A Peer Socket Bpush.send("Part A",zmq.ZMQ_SNDMORE);push.send("Msg2");sendsendsendsend3 Msg6 Part A6 Part B4 Msg2To ATo BTo A
    • PatternsPush / PullBinding the Pull socketPushs outgoing messages are round-robined among all connected peers.push.send("Msg");push.send("PartB");Outgoing Queue Peer Socket A Peer Socket Bpush.send("Part A",zmq.ZMQ_SNDMORE);push.send("Msg2");sendsendsendsendsendsendsend3 Msg6 Part A6 Part B4 Msg2To ATo BTo A
    • PatternsPush / PullBinding the Pull socketDisconnections and reconnections are handled automatically.Outgoing Queue Peer Socket A Peer Socket Bsendsend-> Down4 More6 Part 16 Part 25 OtherTo ATo ATo Bpush.send("More"); sendpush.send("Part2");push.send("Part 1",zmq.ZMQ_SNDMORE);sendsendpush.send("Other");send
    • PatternsPush / PullBinding the Pull socketSending is asynchronous, it depends on the availability of the receiver.Outgoing Queue Peer Socket A Peer Socket Bsend-> Up again5 OtherTo Bpush.send("More"); sendpush.send("Part2");push.send("Part 1",zmq.ZMQ_SNDMORE);sendsendpush.send("Other");send
    • PatternsPush / PullØ PULLØ PUSH #1...Ø PUSH #2Active agents / collector pattern
    • PatternsPublish / SubscribeBroadcast data distributionØ SUB #1Ø PUB #1...Ø SUB #2Ø SUB #2
    • Message distributionPatternsPublish / Subscribesock.send("Msg");Outgoing Queue3 MsgSubscriber ASubscriber BsendsendOutgoing messages are send to each connected and available peers.Subscriber C
    • Message distributionPatternsPublish / Subscribesock.send("Msg");Outgoing Queue3 MsgSubscriber ASubscriber BsendsendOutgoing messages are send to each connected and available peers.Subscriber C5 Othersock.send("Other");send
    • Subscribers can register to a specific topic.Data filteringPatternsPublish / SubscribeØMQ PublisherSubscriber ASubscriber BSubscriber COutgoing Queuesubscribe ARsubscribe VESocketsubscribe VE
    • Subscribers can register to a specific topic.Data filteringPatternsPublish / SubscribeØMQ PublisherSubscriber ASubscriber BSubscriber COutgoing Queuesubscribe ARsubscribe VESocketsubscribe VEsock.send("ARnews");send7 AR newssend
    • Subscribers can register to a specific topic.Data filteringPatternsPublish / SubscribeØMQ PublisherSubscriber ASubscriber BSubscriber COutgoing Queuesubscribe ARsubscribe VESocketsubscribe VEsock.send("VEnews");send7 VE newssendsend
    • PatternsPublish / SubscribeData filtering examplepub = zmq.socket(pub);pub.bindSync("tcp://10.0.0.12:3055");count = 0setInterval(function() {pub.send("TEST " + count++);}, 1000);sub = zmq.socket(sub);sub.connect("tcp://10.0.0.12:3055");sub.subscribe("TEST");sub.on("message", function(msg) {console.log("Received: " + msg);});// older messages wont be// received> TEST 6> TEST 7> TEST 8> TEST 9> TEST 10running thismightoutput ->
    • PatternsPublish / SubscribeMulticast examplevar zmq = require(zmq),pub = zmq.socket(pub);pub.bind("epgm://10.0.0.12;239.192.1.1:3055",function(err) {if(err) throw err;})setInterval(function() {pub.send("From pid: " + process.pid);}, 1000);var zmq = require(zmq),sub = zmq.socket(sub);sub.connect("epgm://10.0.0.13;239.192.1.1:3055");sub.subscribe("");sub.on(message, function(msg) {console.log("Received " + msg);});(*) See running demo at http://youtu.be/NQrH0SATPk0
    • "XSUB and XPUB are exactly like SUB and PUB except they exposesubscriptions as special messages."Data distribution proxyPatternsXSUB / XPUBXSUB/XPUB ProxySubscriber ASubscriber BSubscriber CXPUB SocketXSUB SocketPublisher APublisher BPublisher CsubscribePYforwardsubscription
    • Messages are forwarded by the proxy to the registered subscribers.Data distribution proxyPatternsXSUB / XPUBXSUB/XPUB ProxySubscriber ASubscriber BSubscriber CXPUB SocketXSUB SocketPublisher APublisher BPublisher Cforwardmessagesend "PY<msg>"send
    • 5. Useful links[ ]
    • Useful LinksWorkshop exampleshttp://github.com/krakatoa/node_zmq_workshop
    • Useful LinksThe Guidehttp://zguide.zeromq.org/page:all
    • Useful LinksPieter Hintjens personal bloghttp://hintjens.com/
    • Useful LinksAPI Referenceshttp://api.zeromq.org/3-2:zmq-sockethttp://api.zeromq.org/3-2:zmq-setsockopt
    • Preguntas ?fedario@gmail.comgithub.com/krakatoaFernando Dario AlonsoGracias !