1
going event driven
+ kafka a rabbit-mq
java group #13 bratislava @danielharcek
2
Agenda
1. Case study
2. Retrospektiva
3. Event-driven(ness)
4. Messaging
5. JMS
6. RMQ
7. Kafka
8. Otazky?
3
Pred
l b
w
e
b
rt
rt
rt
rt
rt
rt
r loader
p loader
MySQL
req
multithreaded
daemons
MySQL
prof
I/O
I/O
read
I/O
mov
I/O
m...
4
Po
l b
w
e
b
rt
rt
rt
rt
rt
rt
rabbit
MySQL
req
arch
MySQL
prof
I/O
read
networ
k
I/O
write
networ
k
I/O
write
networ
k
...
5
#pred_a_po
rt
rt
rt
rt
rt
rt
r loader
p loader
MySQL
req
MySQL
prof
reports
campaigns
reports
aggregate
reports
custom
r...
6
Co sme ziskali
• usetrili sme VELA disk i/o
• usetrili load na DB a zredukovali mnozstvo DB
hostov (zostal vlastne uz ib...
7
“Reaktivita” je dnes proste
KAJŠMENTKE
8
Event-driven(ness)
But now a new architecture has evolved to let developers
conceptualize and build applications that sa...
9
Potreba
Komplexne systemy pracujuce s tokmi
velkeho objemu dat s potrebou
odozvy v takmer realnom case,
“neustalym” upti...
10
Messaging 101
Push based (zvacsa)
Producer (agent)
Consumer (sink)
“Event consumers subscribe to middleware which recei...
11
Messaging broker / broker-less
O(n2) O(n)
http://www.eaipatterns.com/ramblings/03_hubandspoke.html
12
Messaging broker / broker-less
broker ako adresar distribuovany broker distribuovany adresar
http://zeromq.org/whitepap...
13
Messaging - rozhodujúce kritériá
• potrebna priepustnost (velkost spravy a pocet sprav), latencia
• clustering / HA
• a...
14
Messaging basic patterns: Queue
• point-to-point
• by default FIFO
• message je spracovany PRAVE
jednym consumerom
• lo...
15
Messaging basic patterns: Publish-Subscribe
• hub / broadcast
• sprava od publishera je
forwardovana na vsetkych
subscr...
16
Messaging basic patterns: Request-reply
• ring
• blizke RPC
• asynchronne spracovanie
odpovede
• klientska aplikacia po...
17
JMS
• Prva verzia 1.0.2b v 2001, 2.0 v 2013
• Java Message Service API
• MOM rozhranie (Message Oriented Middleware)
• ...
18
RMQ
• Vyspely broker (komplexne moznosti routovania, workflows), dokumentacia,
tooling
• AMQP +plugins
• Drivre do vset...
19
Kafka
• distribuovany pub-sub messaging system, skor klient-server ako broker (urceny pre
konkretny typ use-casov – spr...
20
Event-driven(ness)
Umoznuje navrhovat systemy, kde
volne previazane komponenty
(asynchronne) komunikuju
prostrednictvom...
21
Otázky?
deh'-ku-yem
za pozornost
a chlapcom z mrkvosoftu za to ze powerpoint NEMA autosave
recovery-save sa nepocita!
Upcoming SlideShare
Loading in …5
×

Going event drive + Kafka a RabbitMQ

311 views

Published on

Kratko o event-driven architekture, zaklady messagingu, porovnanie mq systemov a scenarov pouzitia RabbitMQ a Kafka.

Published in: Data & Analytics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
311
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • este dopln impression transfer
    request-reply (ako tam bol na zaciatku),
  • message isla len jedna uz
    Prerabka nie je uplna (prisposobil som ju tak aby niektore aspekty vynikli)
  • Messaging je vlastne enterprise integration pattern (aj spring integration)
    JMS

    Iba stare myslienky aplikovane do sucasneho kontextu nasadenia a vyzadovaných vlastnosti isteho typu aplikacii.
    Architektura zalozena na vytvarani, detekovani / notifikovani, konzumovani a reagovani na udalosti.
    Udalost je zmena stavu. Zmena stavu emituje (asynchronnu) spravu – notifikaciu o udalosti.
    Je komplementárna so SOA.

  • Going event drive + Kafka a RabbitMQ

    1. 1. 1 going event driven + kafka a rabbit-mq java group #13 bratislava @danielharcek
    2. 2. 2 Agenda 1. Case study 2. Retrospektiva 3. Event-driven(ness) 4. Messaging 5. JMS 6. RMQ 7. Kafka 8. Otazky?
    3. 3. 3 Pred l b w e b rt rt rt rt rt rt r loader p loader MySQL req multithreaded daemons MySQL prof I/O I/O read I/O mov I/O mov networ k I/O write networ k I/O write networ k reports campaigns MySQL ad MySQL sys reports aggregate reports custom reports visiitors w e b sync tab delimited files chunked per minute mostly cron-scheduled once per day networ k I/O write MySQL req archive
    4. 4. 4 Po l b w e b rt rt rt rt rt rt rabbit MySQL req arch MySQL prof I/O read networ k I/O write networ k I/O write networ k reports campaigns MySQL ad MySQL sys reports aggregate reports custom reports visiitors w e b sync request sent, persisted and forwarded over network daemons & batch (cron jobs) networ k I/O write
    5. 5. 5 #pred_a_po rt rt rt rt rt rt r loader p loader MySQL req MySQL prof reports campaigns reports aggregate reports custom reports visiitors rabbit reports campaigns reports aggregate reports custom reports visiitors rt rt rt rt rt rt v e r s u s
    6. 6. 6 Co sme ziskali • usetrili sme VELA disk i/o • usetrili load na DB a zredukovali mnozstvo DB hostov (zostal vlastne uz iba “archiv” requestov) • zlepsili distribuovatelnost • reporty sa stali menej previazane (netrebalo zdielany diskovy caching) • moznost jednoduchsie pridavat nove typy worker reportov • moznost signalovat potrebu recountu • skoro-real time countre (predtym komplikovane koli loadu na DB) • ak nam padol tracker tak sme nestratili (takmer) ziadne requesty • zjednodusila sa nam distribucia a mohli sme sa viac sustredit na optimalizaciu reportingu • ak padol report spravy si nan pockaju (to ale nebol problem predtym) rabbit reports campaigns reports aggregate reports custom reports visiitors rt rt rt rt rt rt
    7. 7. 7 “Reaktivita” je dnes proste KAJŠMENTKE
    8. 8. 8 Event-driven(ness) But now a new architecture has evolved to let developers conceptualize and build applications that satisfy today’s demands. We call these Reactive Applications. This architecture allows developers to build systems that are event-driven, scalable, resilient and responsive. http://www.reactivemanifesto.org/ NIC NOVE!
    9. 9. 9 Potreba Komplexne systemy pracujuce s tokmi velkeho objemu dat s potrebou odozvy v takmer realnom case, “neustalym” uptimom nasadzovane do cloudoveho prostredia s flexibilnym modelom skalovania a spravy. napr. IoT, mobilne appky, automaticke tradingove systemy
    10. 10. 10 Messaging 101 Push based (zvacsa) Producer (agent) Consumer (sink) “Event consumers subscribe to middleware which receives notification of an event from producer(s).” Vyuziva messaging middleware (backbone) • vacsinou menej alebo viac sofistikovany Broker (message manager) • alebo broker-less framework, jednoducho “Queue”* (p2p, napr. ZMQ) Durability vs. persistence (subscription vs. message) Garancia radenia (ziadna, FIFO)* Garancia dorucenia (at-most-once, at-least-once, exactly once)
    11. 11. 11 Messaging broker / broker-less O(n2) O(n) http://www.eaipatterns.com/ramblings/03_hubandspoke.html
    12. 12. 12 Messaging broker / broker-less broker ako adresar distribuovany broker distribuovany adresar http://zeromq.org/whitepapers:brokerless
    13. 13. 13 Messaging - rozhodujúce kritériá • potrebna priepustnost (velkost spravy a pocet sprav), latencia • clustering / HA • aka topologia (broker, nebroker, aky workflow) • perzistencia (mat consumerov ktori nie su pern. online) • moznosti routingu, filtrovania, fransformacie • push a konzumovania batchov sprav • delivery a ordering garancie • “multiplatformovost” (klienti, drivers) a vyspelost • podpora protokolov • ttl, delayed / scheduled messages, prioritizacia messagov • acknowledgment / receipt notification
    14. 14. 14 Messaging basic patterns: Queue • point-to-point • by default FIFO • message je spracovany PRAVE jednym consumerom • logovanie udalosti / monitoring • load leveling (cakaju vo fronte tak ako system stiha) • load balancing (pridanie novych async workerov) producer queue consumer producer queue consumer consumer consumer
    15. 15. 15 Messaging basic patterns: Publish-Subscribe • hub / broadcast • sprava od publishera je forwardovana na vsetkych subscriberov • * topic • propagacia udalosti (napr. MMO, push notifikacie, live updaty skore zapasu a podobne) • integracie producer topic consumer consumer consumer
    16. 16. 16 Messaging basic patterns: Request-reply • ring • blizke RPC • asynchronne spracovanie odpovede • klientska aplikacia posle search query, backend searchne, vysledok sa vrati naspat • aplikacia si vyziada stav inej aplikacie (napr. progress tasku) producer request q consumer reply q
    17. 17. 17 JMS • Prva verzia 1.0.2b v 2001, 2.0 v 2013 • Java Message Service API • MOM rozhranie (Message Oriented Middleware) • Nepopisuje protokol! (teda dve implementacie JMS nemusia vediet komunikovat) • Provider, Client – Producer / Consumer, Message, Queue (per Consumer), Topic (distribucny mechanizmus) • Nema garantovany ordering, dorucenie at-most-once alebo once-and-only- once (ak je message persistentny) • Point-to-point a Publish-Subscribe • Implentacie: ActiveMQ, Qpid, Redis, …
    18. 18. 18 RMQ • Vyspely broker (komplexne moznosti routovania, workflows), dokumentacia, tooling • AMQP +plugins • Drivre do vsetkych relevantnych jazykov a kopec pluginov • Publisher, Exchange, Route, Queue, Consumer • Echanges: Direct, Fanout, Topic, Headers • Exchange su by default transientne, ich durabilitu a persistenciu je treba explicitne deklarovat • Consumeri maju push aj pull API • ACK, a volba kedy poslat, nie je 100%, expiracia atd • pub-sub je Fanout • clustering, federation (plugin) a queues replication • Vhodny ak menej ako 20k+/sec* a ak potreba komplexnych routing scenarov
    19. 19. 19 Kafka • distribuovany pub-sub messaging system, skor klient-server ako broker (urceny pre konkretny typ use-casov – spracovanie real time aktivity streamov, zber metrik, logovanie) • 0.8.x, meni sa pod rukami • klienti pre kazdy relevantny jazyk • messages su persistovane na servri, kafka zapise a potom zisti kto fetchuje, konfigurovatelny “rolling window of time” (cas alebo miesto na hdd), jedna kopia stremu pre N consumerov • dorucenie v poradi a at-least-once garancia • producer-centric (t.j. producer si strazi svoj stav – rmq ma metadata na strane brokera • pull-based (data fetchujeme a mozme aj specifikovat offset – robit rewind) • Cosumer, producer, topic, partition (ak su consumer groups) • Potrebuje Zookeeper – distribuovany register / adresar, je pouzity na koordinaciu clustra producerov, consumerov a “brokera” • Od 0.8 replikacia partitions, partitioning definovany producerom, Kafka rozhoduje ako rozhodi partitions na brokerov, aj ACK • Vhodny ak treba vysoku priepustnost (100k+/sec)*
    20. 20. 20 Event-driven(ness) Umoznuje navrhovat systemy, kde volne previazane komponenty (asynchronne) komunikuju prostrednictvom sprav a takyto dizajn vediet k implementacii ktora zjedodusuje rozsirovanie a spravu systemu. Pouzitie je teda na distribuciu uloh (routing), spravu (management) systemu, transformaciu* a kontrolu (monitoring). Asynchronicita umoznuje efektivne zdielat prostriedky “jedneho HW vlakna” / vypoctovej jednotky resp. komunikacneho kanala. Non- blocking vedie k nizsej latencii a vyssej priepustnosti. Konkurentnost priamo v dizajne.
    21. 21. 21 Otázky? deh'-ku-yem za pozornost a chlapcom z mrkvosoftu za to ze powerpoint NEMA autosave recovery-save sa nepocita!

    ×