SlideShare a Scribd company logo
Distributed app
 development
 node.js + zeroMQ = awwyeah
ZeroMQ from a noob’s
     viewpoint
RUBEN TAN

• NodeJS enthusiast
• Author of eazy
  (https://github.com/soggie/eazy)


• Dota 2-er
DISCLAIMER

• I AM NOT A ZEROMQ EXPERT
• (not even remotely close)
• Case study by consulting a successful
  implementer of ZMQ in an adserver
Objective

• 1,000,000,000 ad impressions per month
  (on average)
• 80ms maximum response time for each ad
• Good luck, Apache
Webserver Choices

• Netty - Java can actually be fast
  https://netty.io


• Gevent - For Python lovers
  http://gevent.org/


• Node.js - The reason why you’re here
  http://nodejs.org/
So... which one?
• All 3, but that’s not the point
• Speed isn’t the most important factor
• Reliability under load is more important
    than pure performance
• 1 missed impression = 1 potential disaster
    (imagine BMW ads showing up in ThePirateBay)

•   (but secretly, I prefer node.js)
Welcome to the
distributed app
   ecosystem
Definitions, definitions

• Distributed: distributed across multiple
  geological “clouds”
• Apps: each app handles a small scope of
  responsibility
• Ecosystem: apps talk to apps via a
  standardized protocol
Distributed

• Route 53 -> HAProxy -> Webserver ->
  Apps
• Scale webservers by creating new instances
  behind HAProxy
• Scale apps by creating new instances in
  their own clouds
Distributed
                   Route 53 (DNS)

                  HAProxy “LB layer”

Webserver            Webserver            Webserver




      App cloud                        App cloud
        (EU)                             (NA)
Apps

• Each app should perform ONE single
  domain of function
• Write the app like you’re writing an API
• The simpler the app, the better
• App can be written in any language
User app

• List of exposed API:
  •   create new user

  •   edit user

  •   delete user

  •   merge user accounts

  •   authenticate user
Ecosystem

• Apps communicate using JSON-RPC
• Apps connect to each other using ZMQ
• “Controller” scripts and monit handles
  lifecycle
Quick & dirty ZeroMQ
• ZMQ = socket library
• Types of socket:
 •   inproc

 •   IPC

 •   TCP

 •   pgm

 •   epgm
Socket Types

• Push-pull
• Req-rep
• Pub-sub (not covered)
• Dealer-router (not covered)
• Xreq-xrep (not covered)
Push-pull
• Push socket pushes data to pull socket(s)
• Pull socket pulls data from push socket(s)

       Push                     Pull
      Socket                   Socket
Push-pull

• Use cases:
 •   Sending email: apps pushes email data to email app

 •   Logging: apps pushes messages to a logging app

 •   Data crunching: apps pushes partials to processing app(s) to work on

 •   Analytics: apps pushes tracking data to analytics app to process
Push-pull

• Push socket can connect to multiple pull
  sockets (fan-out)
• Pull sockets can listen on multiple push
  sockets (fan-in)
• Best part: ZeroMQ handles the load
  balancing (usually via a LRU)
Advanced Push-Pull
• Example: Log file processing
• Log file needs to be parsed
• Push app (ventilator) parses the log file
• Push app continuously fires partials to pull
  apps (workers)
• Workers push into another pull app
  (collector)
Advanced Push-pull
                           Worker




                           Worker
             push                   push
Ventilator                                        Collector
                    pull                   pull
                           Worker




                           Worker
How do we scale?
ADD MORE
WORKERS
Req-rep

• Req socket sends data to rep socket
• Rep socket receives data from req socket
• Rep socket replies to req socket
• NOTE: Req socket always expects
  replies
  (remember that girl that you waited 5 years for?)
Req-rep

• Use cases:
 •   Inter-app communications: apps that has dependencies between
     each other uses a REQ-REP pattern

 •   Caching: apps req cached items from cache app, cache app rep with
     cached item

 •   API: 3rd party developers issue req to your API app, which filters the
     request and replies with the results
Req-rep
 REQ                   REP
Socket                Socket
          req

 REQ                   REP
Socket                Socket



                rep
 REQ                   REP
Socket                Socket
How to scale?
ADD MORE REP APPS
All together now...
Sample ZMQ code:
 01   var zmq = require('zmq'),
 02     push = zmq.socket('push'),
 03     pullA = zmq.socket('pull'),
 04     pullB = zmq.socket('pull');
 05
 06   pullA.on('message', function (msg) {
 07     console.log('pullA received ' + msg);
 08   });
 09
 10   pullB.on('message', function (msg) {
 11     console.log('pullB received ' + msg);
 12   });
 13
 14   pullA.bindSync('inproc://nodehack-rocks');
 15   pullB.bindSync('inproc://oh-la-la');
 16
 17   push.connect('inproc://nodehack-rocks');
 18   push.connect('inproc://oh-la-la');
 19
 20   for (var i = 0; i < 10; i++) {
 21     push.send('this is message ' + i);
 22   }
Notes
• Each app can define and use multiple ZMQ
  sockets
• You can organize your app’s internals using
  inproc ZMQ sockets (better MVC)
• Controller scripts are important
• There are advanced patterns for reliability
  (read the ZMQ guide)
Libraries

• NodeJS ZMQ binding
  (https://github.com/JustinTulloss/zeromq.node)


• ZMQ Guide
  (http://zguide.zeromq.org/page:all)


• Eazy
  (https://github.com/soggie/eazy)
- the end -

More Related Content

What's hot

Overview of ZeroMQ
Overview of ZeroMQOverview of ZeroMQ
Overview of ZeroMQ
pieterh
 
ZeroMQ in PHP
ZeroMQ in PHPZeroMQ in PHP
ZeroMQ
ZeroMQZeroMQ
Lindsay distributed geventzmq
Lindsay distributed geventzmqLindsay distributed geventzmq
Lindsay distributed geventzmq
Robin Xiao
 
Europycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQEuropycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQ
fcrippa
 
Event Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQEvent Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQ
Luke Luo
 
ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!
Pedro Januário
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uring
ScyllaDB
 
CurveZMQ, ZMTP and other Dubious Characters
CurveZMQ, ZMTP and other Dubious CharactersCurveZMQ, ZMTP and other Dubious Characters
CurveZMQ, ZMTP and other Dubious Characters
pieterh
 
Skydive 5/07/2016
Skydive 5/07/2016Skydive 5/07/2016
Skydive 5/07/2016
Sylvain Afchain
 
Zero mq logs
Zero mq logsZero mq logs
Zero mq logs
Tomas Doran
 
Skydive, real-time network analyzer, container integration
Skydive, real-time network analyzer, container integrationSkydive, real-time network analyzer, container integration
Skydive, real-time network analyzer, container integration
Sylvain Afchain
 
Skydive, real-time network analyzer
Skydive, real-time network analyzer Skydive, real-time network analyzer
Skydive, real-time network analyzer
Sylvain Afchain
 
Skydive 31 janv. 2016
Skydive 31 janv. 2016Skydive 31 janv. 2016
Skydive 31 janv. 2016
Sylvain Afchain
 
The art of concurrent programming
The art of concurrent programmingThe art of concurrent programming
The art of concurrent programming
Iskren Chernev
 
Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?
ScyllaDB
 
Rust Primer
Rust PrimerRust Primer
Rust Primer
Knoldus Inc.
 
Netty @Apple: Large Scale Deployment/Connectivity
Netty @Apple: Large Scale Deployment/ConnectivityNetty @Apple: Large Scale Deployment/Connectivity
Netty @Apple: Large Scale Deployment/Connectivity
C4Media
 
Scapy talk
Scapy talkScapy talk
Practical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profilingPractical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profiling
Lubomir Rintel
 

What's hot (20)

Overview of ZeroMQ
Overview of ZeroMQOverview of ZeroMQ
Overview of ZeroMQ
 
ZeroMQ in PHP
ZeroMQ in PHPZeroMQ in PHP
ZeroMQ in PHP
 
ZeroMQ
ZeroMQZeroMQ
ZeroMQ
 
Lindsay distributed geventzmq
Lindsay distributed geventzmqLindsay distributed geventzmq
Lindsay distributed geventzmq
 
Europycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQEuropycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQ
 
Event Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQEvent Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQ
 
ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uring
 
CurveZMQ, ZMTP and other Dubious Characters
CurveZMQ, ZMTP and other Dubious CharactersCurveZMQ, ZMTP and other Dubious Characters
CurveZMQ, ZMTP and other Dubious Characters
 
Skydive 5/07/2016
Skydive 5/07/2016Skydive 5/07/2016
Skydive 5/07/2016
 
Zero mq logs
Zero mq logsZero mq logs
Zero mq logs
 
Skydive, real-time network analyzer, container integration
Skydive, real-time network analyzer, container integrationSkydive, real-time network analyzer, container integration
Skydive, real-time network analyzer, container integration
 
Skydive, real-time network analyzer
Skydive, real-time network analyzer Skydive, real-time network analyzer
Skydive, real-time network analyzer
 
Skydive 31 janv. 2016
Skydive 31 janv. 2016Skydive 31 janv. 2016
Skydive 31 janv. 2016
 
The art of concurrent programming
The art of concurrent programmingThe art of concurrent programming
The art of concurrent programming
 
Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?
 
Rust Primer
Rust PrimerRust Primer
Rust Primer
 
Netty @Apple: Large Scale Deployment/Connectivity
Netty @Apple: Large Scale Deployment/ConnectivityNetty @Apple: Large Scale Deployment/Connectivity
Netty @Apple: Large Scale Deployment/Connectivity
 
Scapy talk
Scapy talkScapy talk
Scapy talk
 
Practical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profilingPractical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profiling
 

Viewers also liked

Introduction to ZeroMQ
Introduction to ZeroMQIntroduction to ZeroMQ
Introduction to ZeroMQ
YiHung Lee
 
Scala and ZeroMQ: Events beyond the JVM
Scala and ZeroMQ: Events beyond the JVMScala and ZeroMQ: Events beyond the JVM
Scala and ZeroMQ: Events beyond the JVM
RUDDER
 
Case study: Insegnalo
Case study: InsegnaloCase study: Insegnalo
Case study: Insegnalo
Giorgio Sironi
 
Case study: iTunes for K-12
Case study: iTunes for K-12Case study: iTunes for K-12
Case study: iTunes for K-12
Giorgio Sironi
 
CouchDB @ PoliMi
CouchDB @ PoliMiCouchDB @ PoliMi
CouchDB @ PoliMi
Giorgio Sironi
 
Chansonnier: web application for multimedia search on song videos
Chansonnier: web application for multimedia search on song videosChansonnier: web application for multimedia search on song videos
Chansonnier: web application for multimedia search on song videos
Giorgio Sironi
 
Queue System and Zend\Queue implementation
Queue System and Zend\Queue implementationQueue System and Zend\Queue implementation
Queue System and Zend\Queue implementation
Gianluca Arbezzano
 
Queue your work
Queue your workQueue your work
Queue your work
Jurian Sluiman
 
Faster PHP apps using Queues and Workers
Faster PHP apps using Queues and WorkersFaster PHP apps using Queues and Workers
Faster PHP apps using Queues and Workers
Richard Baker
 
Blind detection of image manipulation @ PoliMi
Blind detection of image manipulation @ PoliMiBlind detection of image manipulation @ PoliMi
Blind detection of image manipulation @ PoliMi
Giorgio Sironi
 
PHP and node.js Together
PHP and node.js TogetherPHP and node.js Together
PHP and node.js Together
Chris Tankersley
 
Building Scalable, Highly Concurrent & Fault Tolerant Systems - Lessons Learned
Building Scalable, Highly Concurrent & Fault Tolerant Systems -  Lessons LearnedBuilding Scalable, Highly Concurrent & Fault Tolerant Systems -  Lessons Learned
Building Scalable, Highly Concurrent & Fault Tolerant Systems - Lessons Learned
Jonas Bonér
 
Comet with node.js and V8
Comet with node.js and V8Comet with node.js and V8
Comet with node.js and V8
amix3k
 
Sistema de Mensajeria de Colas con ZeroMQ y Python
Sistema de Mensajeria de Colas con ZeroMQ y PythonSistema de Mensajeria de Colas con ZeroMQ y Python
Sistema de Mensajeria de Colas con ZeroMQ y Python
Ernesto Crespo
 
Distributed Queue System using Gearman
Distributed Queue System using GearmanDistributed Queue System using Gearman
Distributed Queue System using Gearman
Eric Cho
 
Case study: Khan Academy
Case study: Khan AcademyCase study: Khan Academy
Case study: Khan Academy
Giorgio Sironi
 
Software Architecture over ZeroMQ
Software Architecture over ZeroMQSoftware Architecture over ZeroMQ
Software Architecture over ZeroMQ
pieterh
 
ZeroMQ Is The Answer
ZeroMQ Is The AnswerZeroMQ Is The Answer
ZeroMQ Is The Answer
Ian Barber
 
Navigation system for blind using GPS & GSM
Navigation system for blind using GPS & GSMNavigation system for blind using GPS & GSM
Navigation system for blind using GPS & GSM
Prateek Anand
 
Map Projections, Datums, GIS and GPS for Everyone
Map Projections, Datums, GIS and GPS for EveryoneMap Projections, Datums, GIS and GPS for Everyone
Map Projections, Datums, GIS and GPS for Everyone
Dr. Geophysics
 

Viewers also liked (20)

Introduction to ZeroMQ
Introduction to ZeroMQIntroduction to ZeroMQ
Introduction to ZeroMQ
 
Scala and ZeroMQ: Events beyond the JVM
Scala and ZeroMQ: Events beyond the JVMScala and ZeroMQ: Events beyond the JVM
Scala and ZeroMQ: Events beyond the JVM
 
Case study: Insegnalo
Case study: InsegnaloCase study: Insegnalo
Case study: Insegnalo
 
Case study: iTunes for K-12
Case study: iTunes for K-12Case study: iTunes for K-12
Case study: iTunes for K-12
 
CouchDB @ PoliMi
CouchDB @ PoliMiCouchDB @ PoliMi
CouchDB @ PoliMi
 
Chansonnier: web application for multimedia search on song videos
Chansonnier: web application for multimedia search on song videosChansonnier: web application for multimedia search on song videos
Chansonnier: web application for multimedia search on song videos
 
Queue System and Zend\Queue implementation
Queue System and Zend\Queue implementationQueue System and Zend\Queue implementation
Queue System and Zend\Queue implementation
 
Queue your work
Queue your workQueue your work
Queue your work
 
Faster PHP apps using Queues and Workers
Faster PHP apps using Queues and WorkersFaster PHP apps using Queues and Workers
Faster PHP apps using Queues and Workers
 
Blind detection of image manipulation @ PoliMi
Blind detection of image manipulation @ PoliMiBlind detection of image manipulation @ PoliMi
Blind detection of image manipulation @ PoliMi
 
PHP and node.js Together
PHP and node.js TogetherPHP and node.js Together
PHP and node.js Together
 
Building Scalable, Highly Concurrent & Fault Tolerant Systems - Lessons Learned
Building Scalable, Highly Concurrent & Fault Tolerant Systems -  Lessons LearnedBuilding Scalable, Highly Concurrent & Fault Tolerant Systems -  Lessons Learned
Building Scalable, Highly Concurrent & Fault Tolerant Systems - Lessons Learned
 
Comet with node.js and V8
Comet with node.js and V8Comet with node.js and V8
Comet with node.js and V8
 
Sistema de Mensajeria de Colas con ZeroMQ y Python
Sistema de Mensajeria de Colas con ZeroMQ y PythonSistema de Mensajeria de Colas con ZeroMQ y Python
Sistema de Mensajeria de Colas con ZeroMQ y Python
 
Distributed Queue System using Gearman
Distributed Queue System using GearmanDistributed Queue System using Gearman
Distributed Queue System using Gearman
 
Case study: Khan Academy
Case study: Khan AcademyCase study: Khan Academy
Case study: Khan Academy
 
Software Architecture over ZeroMQ
Software Architecture over ZeroMQSoftware Architecture over ZeroMQ
Software Architecture over ZeroMQ
 
ZeroMQ Is The Answer
ZeroMQ Is The AnswerZeroMQ Is The Answer
ZeroMQ Is The Answer
 
Navigation system for blind using GPS & GSM
Navigation system for blind using GPS & GSMNavigation system for blind using GPS & GSM
Navigation system for blind using GPS & GSM
 
Map Projections, Datums, GIS and GPS for Everyone
Map Projections, Datums, GIS and GPS for EveryoneMap Projections, Datums, GIS and GPS for Everyone
Map Projections, Datums, GIS and GPS for Everyone
 

Similar to Distributed app development with nodejs and zeromq

Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10
Vishnu Kannan
 
Hello world - intro to node js
Hello world - intro to node jsHello world - intro to node js
Hello world - intro to node js
Refresh Annapolis Valley
 
Real time web
Real time webReal time web
Real time web
Medhat Dawoud
 
Free Mongo on OpenShift
Free Mongo on OpenShiftFree Mongo on OpenShift
Free Mongo on OpenShift
Steven Pousty
 
AppScale @ LA.rb
AppScale @ LA.rbAppScale @ LA.rb
AppScale @ LA.rb
Chris Bunch
 
OpenShift with Eclipse Tooling - EclipseCon 2012
OpenShift with Eclipse Tooling - EclipseCon 2012OpenShift with Eclipse Tooling - EclipseCon 2012
OpenShift with Eclipse Tooling - EclipseCon 2012
Steven Pousty
 
Reactive Programming in Java 8 with Rx-Java
Reactive Programming in Java 8 with Rx-JavaReactive Programming in Java 8 with Rx-Java
Reactive Programming in Java 8 with Rx-Java
Kasun Indrasiri
 
Advanced Web Technology.pptx
Advanced Web Technology.pptxAdvanced Web Technology.pptx
Advanced Web Technology.pptx
ssuser35fdf2
 
PAC 2019 virtual Bruno Audoux
PAC 2019 virtual Bruno Audoux PAC 2019 virtual Bruno Audoux
PAC 2019 virtual Bruno Audoux
Neotys
 
Operating your Production API
Operating your Production APIOperating your Production API
Operating your Production API
Amazon Web Services
 
Beginners Node.js
Beginners Node.jsBeginners Node.js
Beginners Node.js
Khaled Mosharraf
 
Adding Real-time Features to PHP Applications
Adding Real-time Features to PHP ApplicationsAdding Real-time Features to PHP Applications
Adding Real-time Features to PHP Applications
Ronny López
 
introduction to node.js
introduction to node.jsintroduction to node.js
introduction to node.js
orkaplan
 
IBM Bluemix OpenWhisk: Serverless Conference 2016, London, UK: The Future of ...
IBM Bluemix OpenWhisk: Serverless Conference 2016, London, UK: The Future of ...IBM Bluemix OpenWhisk: Serverless Conference 2016, London, UK: The Future of ...
IBM Bluemix OpenWhisk: Serverless Conference 2016, London, UK: The Future of ...
OpenWhisk
 
Kubernetes Architecture - beyond a black box - Part 2
Kubernetes Architecture - beyond a black box - Part 2Kubernetes Architecture - beyond a black box - Part 2
Kubernetes Architecture - beyond a black box - Part 2
Hao H. Zhang
 
Performance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data PlatformsPerformance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data Platforms
DataWorks Summit/Hadoop Summit
 
Erlang factory 2011 london
Erlang factory 2011 londonErlang factory 2011 london
Erlang factory 2011 london
Paolo Negri
 
Designing for Scale
Designing for ScaleDesigning for Scale
Designing for Scale
Wooga
 
Intro to CakePHP
Intro to CakePHPIntro to CakePHP
Intro to CakePHP
Walther Lalk
 
How to Build a Big Data Application: Serverless Edition
How to Build a Big Data Application: Serverless EditionHow to Build a Big Data Application: Serverless Edition
How to Build a Big Data Application: Serverless Edition
ecobold
 

Similar to Distributed app development with nodejs and zeromq (20)

Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10Kubernetes deep dive - - Huawei 2015-10
Kubernetes deep dive - - Huawei 2015-10
 
Hello world - intro to node js
Hello world - intro to node jsHello world - intro to node js
Hello world - intro to node js
 
Real time web
Real time webReal time web
Real time web
 
Free Mongo on OpenShift
Free Mongo on OpenShiftFree Mongo on OpenShift
Free Mongo on OpenShift
 
AppScale @ LA.rb
AppScale @ LA.rbAppScale @ LA.rb
AppScale @ LA.rb
 
OpenShift with Eclipse Tooling - EclipseCon 2012
OpenShift with Eclipse Tooling - EclipseCon 2012OpenShift with Eclipse Tooling - EclipseCon 2012
OpenShift with Eclipse Tooling - EclipseCon 2012
 
Reactive Programming in Java 8 with Rx-Java
Reactive Programming in Java 8 with Rx-JavaReactive Programming in Java 8 with Rx-Java
Reactive Programming in Java 8 with Rx-Java
 
Advanced Web Technology.pptx
Advanced Web Technology.pptxAdvanced Web Technology.pptx
Advanced Web Technology.pptx
 
PAC 2019 virtual Bruno Audoux
PAC 2019 virtual Bruno Audoux PAC 2019 virtual Bruno Audoux
PAC 2019 virtual Bruno Audoux
 
Operating your Production API
Operating your Production APIOperating your Production API
Operating your Production API
 
Beginners Node.js
Beginners Node.jsBeginners Node.js
Beginners Node.js
 
Adding Real-time Features to PHP Applications
Adding Real-time Features to PHP ApplicationsAdding Real-time Features to PHP Applications
Adding Real-time Features to PHP Applications
 
introduction to node.js
introduction to node.jsintroduction to node.js
introduction to node.js
 
IBM Bluemix OpenWhisk: Serverless Conference 2016, London, UK: The Future of ...
IBM Bluemix OpenWhisk: Serverless Conference 2016, London, UK: The Future of ...IBM Bluemix OpenWhisk: Serverless Conference 2016, London, UK: The Future of ...
IBM Bluemix OpenWhisk: Serverless Conference 2016, London, UK: The Future of ...
 
Kubernetes Architecture - beyond a black box - Part 2
Kubernetes Architecture - beyond a black box - Part 2Kubernetes Architecture - beyond a black box - Part 2
Kubernetes Architecture - beyond a black box - Part 2
 
Performance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data PlatformsPerformance Comparison of Streaming Big Data Platforms
Performance Comparison of Streaming Big Data Platforms
 
Erlang factory 2011 london
Erlang factory 2011 londonErlang factory 2011 london
Erlang factory 2011 london
 
Designing for Scale
Designing for ScaleDesigning for Scale
Designing for Scale
 
Intro to CakePHP
Intro to CakePHPIntro to CakePHP
Intro to CakePHP
 
How to Build a Big Data Application: Serverless Edition
How to Build a Big Data Application: Serverless EditionHow to Build a Big Data Application: Serverless Edition
How to Build a Big Data Application: Serverless Edition
 

More from Ruben Tan

Basic distributed systems principles
Basic distributed systems principlesBasic distributed systems principles
Basic distributed systems principles
Ruben Tan
 
Demystifying blockchains
Demystifying blockchainsDemystifying blockchains
Demystifying blockchains
Ruben Tan
 
Banking on blockchains
Banking on blockchainsBanking on blockchains
Banking on blockchains
Ruben Tan
 
Consensus in distributed computing
Consensus in distributed computingConsensus in distributed computing
Consensus in distributed computing
Ruben Tan
 
Defensive programming in Javascript and Node.js
Defensive programming in Javascript and Node.jsDefensive programming in Javascript and Node.js
Defensive programming in Javascript and Node.js
Ruben Tan
 
Client-side storage
Client-side storageClient-side storage
Client-side storage
Ruben Tan
 
How we git - commit policy and code review
How we git - commit policy and code reviewHow we git - commit policy and code review
How we git - commit policy and code review
Ruben Tan
 
NodeHack #2 - MVP
NodeHack #2 - MVPNodeHack #2 - MVP
NodeHack #2 - MVP
Ruben Tan
 
40 square's git workflow
40 square's git workflow40 square's git workflow
40 square's git workflow
Ruben Tan
 
Unit testing for 40 square software
Unit testing for 40 square softwareUnit testing for 40 square software
Unit testing for 40 square software
Ruben Tan
 

More from Ruben Tan (10)

Basic distributed systems principles
Basic distributed systems principlesBasic distributed systems principles
Basic distributed systems principles
 
Demystifying blockchains
Demystifying blockchainsDemystifying blockchains
Demystifying blockchains
 
Banking on blockchains
Banking on blockchainsBanking on blockchains
Banking on blockchains
 
Consensus in distributed computing
Consensus in distributed computingConsensus in distributed computing
Consensus in distributed computing
 
Defensive programming in Javascript and Node.js
Defensive programming in Javascript and Node.jsDefensive programming in Javascript and Node.js
Defensive programming in Javascript and Node.js
 
Client-side storage
Client-side storageClient-side storage
Client-side storage
 
How we git - commit policy and code review
How we git - commit policy and code reviewHow we git - commit policy and code review
How we git - commit policy and code review
 
NodeHack #2 - MVP
NodeHack #2 - MVPNodeHack #2 - MVP
NodeHack #2 - MVP
 
40 square's git workflow
40 square's git workflow40 square's git workflow
40 square's git workflow
 
Unit testing for 40 square software
Unit testing for 40 square softwareUnit testing for 40 square software
Unit testing for 40 square software
 

Recently uploaded

Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Wask
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Precisely
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Zilliz
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
HarisZaheer8
 
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process MiningTrusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
LucaBarbaro3
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
ScyllaDB
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Tatiana Kojar
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Jeffrey Haguewood
 
Public CyberSecurity Awareness Presentation 2024.pptx
Public CyberSecurity Awareness Presentation 2024.pptxPublic CyberSecurity Awareness Presentation 2024.pptx
Public CyberSecurity Awareness Presentation 2024.pptx
marufrahmanstratejm
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
saastr
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Alex Pruden
 

Recently uploaded (20)

Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying AheadDigital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and MilvusGenerating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
 
AWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptxAWS Cloud Cost Optimization Presentation.pptx
AWS Cloud Cost Optimization Presentation.pptx
 
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process MiningTrusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-EfficiencyFreshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
 
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
 
Public CyberSecurity Awareness Presentation 2024.pptx
Public CyberSecurity Awareness Presentation 2024.pptxPublic CyberSecurity Awareness Presentation 2024.pptx
Public CyberSecurity Awareness Presentation 2024.pptx
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
Overcoming the PLG Trap: Lessons from Canva's Head of Sales & Head of EMEA Da...
 
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
 

Distributed app development with nodejs and zeromq

  • 1. Distributed app development node.js + zeroMQ = awwyeah
  • 2. ZeroMQ from a noob’s viewpoint
  • 3. RUBEN TAN • NodeJS enthusiast • Author of eazy (https://github.com/soggie/eazy) • Dota 2-er
  • 4. DISCLAIMER • I AM NOT A ZEROMQ EXPERT • (not even remotely close) • Case study by consulting a successful implementer of ZMQ in an adserver
  • 5. Objective • 1,000,000,000 ad impressions per month (on average) • 80ms maximum response time for each ad • Good luck, Apache
  • 6. Webserver Choices • Netty - Java can actually be fast https://netty.io • Gevent - For Python lovers http://gevent.org/ • Node.js - The reason why you’re here http://nodejs.org/
  • 7. So... which one? • All 3, but that’s not the point • Speed isn’t the most important factor • Reliability under load is more important than pure performance • 1 missed impression = 1 potential disaster (imagine BMW ads showing up in ThePirateBay) • (but secretly, I prefer node.js)
  • 9. Definitions, definitions • Distributed: distributed across multiple geological “clouds” • Apps: each app handles a small scope of responsibility • Ecosystem: apps talk to apps via a standardized protocol
  • 10. Distributed • Route 53 -> HAProxy -> Webserver -> Apps • Scale webservers by creating new instances behind HAProxy • Scale apps by creating new instances in their own clouds
  • 11. Distributed Route 53 (DNS) HAProxy “LB layer” Webserver Webserver Webserver App cloud App cloud (EU) (NA)
  • 12. Apps • Each app should perform ONE single domain of function • Write the app like you’re writing an API • The simpler the app, the better • App can be written in any language
  • 13. User app • List of exposed API: • create new user • edit user • delete user • merge user accounts • authenticate user
  • 14. Ecosystem • Apps communicate using JSON-RPC • Apps connect to each other using ZMQ • “Controller” scripts and monit handles lifecycle
  • 15. Quick & dirty ZeroMQ • ZMQ = socket library • Types of socket: • inproc • IPC • TCP • pgm • epgm
  • 16. Socket Types • Push-pull • Req-rep • Pub-sub (not covered) • Dealer-router (not covered) • Xreq-xrep (not covered)
  • 17. Push-pull • Push socket pushes data to pull socket(s) • Pull socket pulls data from push socket(s) Push Pull Socket Socket
  • 18. Push-pull • Use cases: • Sending email: apps pushes email data to email app • Logging: apps pushes messages to a logging app • Data crunching: apps pushes partials to processing app(s) to work on • Analytics: apps pushes tracking data to analytics app to process
  • 19. Push-pull • Push socket can connect to multiple pull sockets (fan-out) • Pull sockets can listen on multiple push sockets (fan-in) • Best part: ZeroMQ handles the load balancing (usually via a LRU)
  • 20. Advanced Push-Pull • Example: Log file processing • Log file needs to be parsed • Push app (ventilator) parses the log file • Push app continuously fires partials to pull apps (workers) • Workers push into another pull app (collector)
  • 21. Advanced Push-pull Worker Worker push push Ventilator Collector pull pull Worker Worker
  • 22. How do we scale?
  • 24. Req-rep • Req socket sends data to rep socket • Rep socket receives data from req socket • Rep socket replies to req socket • NOTE: Req socket always expects replies (remember that girl that you waited 5 years for?)
  • 25. Req-rep • Use cases: • Inter-app communications: apps that has dependencies between each other uses a REQ-REP pattern • Caching: apps req cached items from cache app, cache app rep with cached item • API: 3rd party developers issue req to your API app, which filters the request and replies with the results
  • 26. Req-rep REQ REP Socket Socket req REQ REP Socket Socket rep REQ REP Socket Socket
  • 28. ADD MORE REP APPS
  • 30. Sample ZMQ code: 01 var zmq = require('zmq'), 02 push = zmq.socket('push'), 03 pullA = zmq.socket('pull'), 04 pullB = zmq.socket('pull'); 05 06 pullA.on('message', function (msg) { 07 console.log('pullA received ' + msg); 08 }); 09 10 pullB.on('message', function (msg) { 11 console.log('pullB received ' + msg); 12 }); 13 14 pullA.bindSync('inproc://nodehack-rocks'); 15 pullB.bindSync('inproc://oh-la-la'); 16 17 push.connect('inproc://nodehack-rocks'); 18 push.connect('inproc://oh-la-la'); 19 20 for (var i = 0; i < 10; i++) { 21 push.send('this is message ' + i); 22 }
  • 31. Notes • Each app can define and use multiple ZMQ sockets • You can organize your app’s internals using inproc ZMQ sockets (better MVC) • Controller scripts are important • There are advanced patterns for reliability (read the ZMQ guide)
  • 32. Libraries • NodeJS ZMQ binding (https://github.com/JustinTulloss/zeromq.node) • ZMQ Guide (http://zguide.zeromq.org/page:all) • Eazy (https://github.com/soggie/eazy)

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n
  31. \n
  32. \n
  33. \n