This document discusses event-driven architectures and messaging. It provides an agenda that includes a case study, retrospective, definitions of event-driven systems and messaging, and an overview of JMS, RabbitMQ and Kafka. Benefits of moving to an event-driven system with RabbitMQ are outlined, including reduced I/O load, improved scalability and distribution. Key criteria for choosing a messaging system include throughput, clustering, topologies, persistence, routing and delivery guarantees. Common messaging patterns like queues, publish-subscribe and request-reply are described. The document concludes with questions.

1. 1
going event driven
+ kafka a rabbit-mq
java group #13 bratislava @danielharcek

2. 2
Agenda
1. Case study
2. Retrospektiva
3. Event-driven(ness)
4. Messaging
5. JMS
6. RMQ
7. Kafka
8. Otazky?

3. 3
Pred
l b
w
e
b
rt
rt
rt
rt
rt
rt
r loader
p loader
MySQL
req
multithreaded
daemons
MySQL
prof
I/O
I/O
read
I/O
mov
I/O
mov
networ
k
I/O
write
networ
k
I/O
write
networ
k
reports
campaigns
MySQL
ad
MySQL
sys
reports
aggregate
reports
custom
reports
visiitors
w
e
b
sync
tab delimited files chunked per minute
mostly cron-scheduled once per day
networ
k I/O
write
MySQL
req
archive

4. 4
Po
l b
w
e
b
rt
rt
rt
rt
rt
rt
rabbit
MySQL
req
arch
MySQL
prof
I/O
read
networ
k
I/O
write
networ
k
I/O
write
networ
k
reports
campaigns
MySQL
ad
MySQL
sys
reports
aggregate
reports
custom
reports
visiitors
w
e
b
sync
request sent,
persisted and
forwarded over
network
daemons & batch (cron jobs)
networ
k I/O
write

5. 5
#pred_a_po
rt
rt
rt
rt
rt
rt
r loader
p loader
MySQL
req
MySQL
prof
reports
campaigns
reports
aggregate
reports
custom
reports
visiitors
rabbit
reports
campaigns
reports
aggregate
reports
custom
reports
visiitors
rt
rt
rt
rt
rt
rt
v
e
r
s
u
s

6. 6
Co sme ziskali
• usetrili sme VELA disk i/o
• usetrili load na DB a zredukovali mnozstvo DB
hostov (zostal vlastne uz iba “archiv”
requestov)
• zlepsili distribuovatelnost
• reporty sa stali menej previazane (netrebalo
zdielany diskovy caching)
• moznost jednoduchsie pridavat nove typy
worker reportov
• moznost signalovat potrebu recountu
• skoro-real time countre (predtym
komplikovane koli loadu na DB)
• ak nam padol tracker tak sme nestratili
(takmer) ziadne requesty
• zjednodusila sa nam distribucia a mohli sme
sa viac sustredit na optimalizaciu reportingu
• ak padol report spravy si nan pockaju (to ale
nebol problem predtym)
rabbit
reports
campaigns
reports
aggregate
reports
custom
reports
visiitors
rt
rt
rt
rt
rt
rt

7. 7
“Reaktivita” je dnes proste
KAJŠMENTKE

8. 8
Event-driven(ness)
But now a new architecture has evolved to let developers
conceptualize and build applications that satisfy today’s
demands.
We call these Reactive Applications. This architecture
allows developers to build systems that are event-driven,
scalable, resilient and responsive.
http://www.reactivemanifesto.org/
NIC NOVE!

9. 9
Potreba
Komplexne systemy pracujuce s tokmi
velkeho objemu dat s potrebou
odozvy v takmer realnom case,
“neustalym” uptimom nasadzovane do
cloudoveho prostredia s flexibilnym
modelom skalovania a spravy.
napr. IoT, mobilne appky, automaticke tradingove systemy

10. 10
Messaging 101
Push based (zvacsa)
Producer (agent)
Consumer (sink)
“Event consumers subscribe to middleware which receives notification of an
event from producer(s).”
Vyuziva messaging middleware (backbone)
• vacsinou menej alebo viac sofistikovany Broker (message manager)
• alebo broker-less framework, jednoducho “Queue”* (p2p, napr. ZMQ)
Durability vs. persistence (subscription vs. message)
Garancia radenia (ziadna, FIFO)*
Garancia dorucenia (at-most-once, at-least-once, exactly once)

11. 11
Messaging broker / broker-less
O(n2) O(n)
http://www.eaipatterns.com/ramblings/03_hubandspoke.html

12. 12
Messaging broker / broker-less
broker ako adresar distribuovany broker distribuovany adresar
http://zeromq.org/whitepapers:brokerless

13. 13
Messaging - rozhodujúce kritériá
• potrebna priepustnost (velkost spravy a pocet sprav), latencia
• clustering / HA
• aka topologia (broker, nebroker, aky workflow)
• perzistencia (mat consumerov ktori nie su pern. online)
• moznosti routingu, filtrovania, fransformacie
• push a konzumovania batchov sprav
• delivery a ordering garancie
• “multiplatformovost” (klienti, drivers) a vyspelost
• podpora protokolov
• ttl, delayed / scheduled messages, prioritizacia messagov
• acknowledgment / receipt notification

14. 14
Messaging basic patterns: Queue
• point-to-point
• by default FIFO
• message je spracovany PRAVE
jednym consumerom
• logovanie udalosti / monitoring
• load leveling (cakaju vo fronte tak
ako system stiha)
• load balancing (pridanie novych
async workerov)
producer queue consumer
producer queue consumer
consumer
consumer

15. 15
Messaging basic patterns: Publish-Subscribe
• hub / broadcast
• sprava od publishera je
forwardovana na vsetkych
subscriberov
• * topic
• propagacia udalosti (napr. MMO,
push notifikacie, live updaty skore
zapasu a podobne)
• integracie
producer topic consumer
consumer
consumer

16. 16
Messaging basic patterns: Request-reply
• ring
• blizke RPC
• asynchronne spracovanie
odpovede
• klientska aplikacia posle search
query, backend searchne,
vysledok sa vrati naspat
• aplikacia si vyziada stav inej
aplikacie (napr. progress tasku)
producer
request q
consumer
reply q

17. 17
JMS
• Prva verzia 1.0.2b v 2001, 2.0 v 2013
• Java Message Service API
• MOM rozhranie (Message Oriented Middleware)
• Nepopisuje protokol! (teda dve implementacie JMS nemusia vediet
komunikovat)
• Provider, Client – Producer / Consumer, Message, Queue (per Consumer),
Topic (distribucny mechanizmus)
• Nema garantovany ordering, dorucenie at-most-once alebo once-and-only-
once (ak je message persistentny)
• Point-to-point a Publish-Subscribe
• Implentacie: ActiveMQ, Qpid, Redis, …

18. 18
RMQ
• Vyspely broker (komplexne moznosti routovania, workflows), dokumentacia,
tooling
• AMQP +plugins
• Drivre do vsetkych relevantnych jazykov a kopec pluginov
• Publisher, Exchange, Route, Queue, Consumer
• Echanges: Direct, Fanout, Topic, Headers
• Exchange su by default transientne, ich durabilitu a persistenciu je treba
explicitne deklarovat
• Consumeri maju push aj pull API
• ACK, a volba kedy poslat, nie je 100%, expiracia atd
• pub-sub je Fanout
• clustering, federation (plugin) a queues replication
• Vhodny ak menej ako 20k+/sec* a ak potreba komplexnych routing
scenarov

19. 19
Kafka
• distribuovany pub-sub messaging system, skor klient-server ako broker (urceny pre
konkretny typ use-casov – spracovanie real time aktivity streamov, zber metrik, logovanie)
• 0.8.x, meni sa pod rukami
• klienti pre kazdy relevantny jazyk
• messages su persistovane na servri, kafka zapise a potom zisti kto fetchuje,
konfigurovatelny “rolling window of time” (cas alebo miesto na hdd), jedna kopia stremu pre
N consumerov
• dorucenie v poradi a at-least-once garancia
• producer-centric (t.j. producer si strazi svoj stav – rmq ma metadata na strane brokera
• pull-based (data fetchujeme a mozme aj specifikovat offset – robit rewind)
• Cosumer, producer, topic, partition (ak su consumer groups)
• Potrebuje Zookeeper – distribuovany register / adresar, je pouzity na koordinaciu clustra
producerov, consumerov a “brokera”
• Od 0.8 replikacia partitions, partitioning definovany producerom, Kafka rozhoduje ako
rozhodi partitions na brokerov, aj ACK
• Vhodny ak treba vysoku priepustnost (100k+/sec)*

20. 20
Event-driven(ness)
Umoznuje navrhovat systemy, kde
volne previazane komponenty
(asynchronne) komunikuju
prostrednictvom sprav a takyto
dizajn vediet k implementacii ktora
zjedodusuje rozsirovanie a spravu
systemu.
Pouzitie je teda na distribuciu uloh
(routing), spravu (management)
systemu, transformaciu* a kontrolu
(monitoring).
Asynchronicita umoznuje efektivne
zdielat prostriedky “jedneho HW
vlakna” / vypoctovej jednotky resp.
komunikacneho kanala. Non-
blocking vedie k nizsej latencii a
vyssej priepustnosti. Konkurentnost
priamo v dizajne.

21. 21
Otázky?
deh'-ku-yem
za pozornost
a chlapcom z mrkvosoftu za to ze powerpoint NEMA autosave
recovery-save sa nepocita!