Europycon2011: Implementing distributed application using ZeroMQ

Implementing distributed applications with...and some other bad guys... Crippa Francesco

Scalability vs Complexity

Scalability vs Complexity what we want Complexity1 Thread 2 Threads 2 Nodes 4 Nodes

Scalability vs Complexity reality Complexity1 Thread 2 Threads 2 Nodes 4 Nodes

The Q in ØMQ

The Ø in ØMQ

The Ø in ØMQZero Broker

The Ø in ØMQZero BrokerZero Latency (as close as possible...)

The Ø in ØMQZero BrokerZero Latency (as close as possible...)Zero administration

The Ø in ØMQZero BrokerZero Latency (as close as possible...)Zero administrationZero cost

The Ø in ØMQZero BrokerZero Latency (as close as possible...)Zero administrationZero costZero waste

Sockets

SocketsUnicast transports (inproc, ipc, tcp)

SocketsUnicast transports (inproc, ipc, tcp)Multicast transports (pgm or epgm)

SocketsUnicast transports (inproc, ipc, tcp)Multicast transports (pgm or epgm)connect() and bind() are independent

SocketsUnicast transports (inproc, ipc, tcp)Multicast transports (pgm or epgm)connect() and bind() are independentThey are asynchronous (with queues)

SocketsUnicast transports (inproc, ipc, tcp)Multicast transports (pgm or epgm)connect() and bind() are independentThey are asynchronous (with queues)They express a certain "messaging pattern"

SocketsUnicast transports (inproc, ipc, tcp)Multicast transports (pgm or epgm)connect() and bind() are independentThey are asynchronous (with queues)They express a certain "messaging pattern"They are not necessarily one-to-one

...and, of course...Cross Platform (Linux, Windows, Mac, etc...)Multiple Languages (c, c++, python, java,ruby, erlang, php, perl, ada, c#, lua, scala,objective-c, go, haskell, racket, cl, basic...)OpenSource

if you have a laptop...http://www.zeromq.org/http://zguide.zeromq.org/

Basic Message Patterns

Server

Client

Server

Client

Publisher SubscriberThe PUB-SUB socket pair is asynchronouswhen you use a SUB socket you must set asubscription using zmq_setsockopt andSUBSCRIBE"slow joiner" symptom

Ventilator

Worker

WorkersAlways synchronize the start of the batchThe ventilators PUSH socket distributestasks to workers (load balancing)The sinks PULL socket collects results fromworkers evenly (fair-queuing)

Fair-Queuing

Sendmessages back

Allowed PatternsPUB and SUB DEALER and DEALERREQ and REP ROUTER and ROUTERREQ and ROUTER PUSH and PULLDEALER and REP PAIR and PAIRDEALER and ROUTER

Scalability

A Publish-Subscribe Proxy

Built-in DevicesQUEUE (request-reply broker.)FORWARDER (pub-sub proxy server)STREAMER (like FORWARDER but forpipeline ﬂows)

Dynamic Scalability...

Dynamic Scalability... SCA LOW LAB ILI !!!! TY

Dynamic Scalability...

ØMQ : Sockets = Python : C++

Europycon2011: Implementing distributed application using ZeroMQ

by fcrippa on Jul 18, 2011

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 198

Statistics

Europycon2011: Implementing distributed application using ZeroMQ Presentation Transcript

http://lanyrd.com	195
https://twitter.com	3