Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
UNDER THE HOODMike Schonberg
Agenda   Transports   Queues   Publishers   Events   Value Added Components   Code   Introduction to Bridges
Transports   Establish connections connections between    publishers and subscribers     Mechanism varies widely between...
Queues   Queue based threading model     One dispatch thread per queue     Several dispatch modes      • Single event: ...
Publishers   Send messages and requests   One publisher per topic   Associated with a specific transport   Thinnest po...
Subscriptions   Subscription to a single topic     Receive message from all publishers of topic   onMsg() callback invo...
Inbox   Receives replies through onMsg() callback     Multiple replies possible      • From different subscribers      •...
Timers   Repeating timers   Interval is floating point in seconds     Resolution dependent on native middleware or OS ...
IO Events   Asynchronous IO support     For non-blocking file descriptors and sockets   Invokes callbacks for read and ...
OpenMAMA: Value-added Components
Other Components   Stateful market data subscriptions/data    quality   Advanced publishing framework   Data dictionary...
Stateful Subscription   Initial value    delivered point-to-    point   Complex sequence    number logic   Other potent...
Data Quality   Essential for Stateful/Market Data Subscriptions   Sequence number monitoring     Request recap on gap d...
Advanced Publishing   Framework for publishing to stateful/market data    subscribers   Sequence number and status field...
Data Dictionary   OpenMAMA supports self describing messages     Field type, name, id (integer) and value   Sending nam...
Fault Tolerance   Applications can run in fault tolerant groups   Every instance has a unique weight   Heart beat messa...
OpenMAMA in Action
Let’s write a program!   Create a publisher     Open a session     Pack a payload     Send message   Create a subscri...
OpenMAMA Publisher Example   Taken from                   int main(void) {    mamapublisherc.c example        initializeM...
Initialization                                     void initializeMama(void) {   Load middleware                    mama_...
Create a Timer                           static void createIntervalTimer(void){   Initialize timer         status = mamaT...
Create a Subscription                            static void createInboundSubscription (void){   Subscription for        ...
Publishing   Publisher requires transport and    static void createPublisher(void){                                      ...
What Does it Do?   Timer callback fired every second       Sends a message on “MAMA_TOPIC”   Listens for messages on “R...
Middleware Bridges - Overview
Bridges   Payload and Message Bridges     Directly reflect top-level API      • Messages, subscriptions, queues, timers,...
mschonberg@nyx.com
Upcoming SlideShare
Loading in …5
×

OpenMAMA Under the Hood

1,974 views

Published on

Published in: Technology, Business
  • Be the first to comment

OpenMAMA Under the Hood

  1. 1. UNDER THE HOODMike Schonberg
  2. 2. Agenda Transports Queues Publishers Events Value Added Components Code Introduction to Bridges
  3. 3. Transports Establish connections connections between publishers and subscribers  Mechanism varies widely between middlewares • NSD (NYSE Technologies Data Fabric) • Broker based (Avis, AMQP, TIBCO Rendezvous) • Multicast topic resolution (Informatica LBM) Global configuration and tuning  Buffer sizes  Network parameters  Throttle rates
  4. 4. Queues Queue based threading model  One dispatch thread per queue  Several dispatch modes • Single event: invoke one callback and return • Timed: dispatch until empty or timeout • Continuous: dispatch forever Callback based Advanced features  Watermarks to monitor queue depth  Enqueue callback to integrate with other event loops Custom callbacks
  5. 5. Publishers Send messages and requests One publisher per topic Associated with a specific transport Thinnest possible wrapper to native publishing Support throttled publishing Advanced publishing framework  dqpublisher in source  Stateful (market data publishing)
  6. 6. Subscriptions Subscription to a single topic  Receive message from all publishers of topic onMsg() callback invoked for each message Creation throttled Optional wildcard support Basic vs. stateful (market data)  Stateful receive initial state  Stateful monitor sequence numbers
  7. 7. Inbox Receives replies through onMsg() callback  Multiple replies possible • From different subscribers • From the same subscriber Parameter to publisher_sendFromInbox() Supports multiple requests
  8. 8. Timers Repeating timers Interval is floating point in seconds  Resolution dependent on native middleware or OS onTimerCallback()  Enqueued on most once per interval  Queue depth makes non-deterministic OpenMAMA provides default implementation
  9. 9. IO Events Asynchronous IO support  For non-blocking file descriptors and sockets Invokes callbacks for read and write readiness Support is optional Integrates asynchronous IO into OpenMAMA event dispatching
  10. 10. OpenMAMA: Value-added Components
  11. 11. Other Components Stateful market data subscriptions/data quality Advanced publishing framework Data dictionary Fault tolerance
  12. 12. Stateful Subscription Initial value delivered point-to- point Complex sequence number logic Other potential applications
  13. 13. Data Quality Essential for Stateful/Market Data Subscriptions Sequence number monitoring  Request recap on gap detection  Recap is an initial image Pre-initial spooling Service disruptions from native middleware Callbacks inform application of data quality events  STALE -> data definitely lost  POSSIBLY_STALE -> data may be lost
  14. 14. Advanced Publishing Framework for publishing to stateful/market data subscribers Sequence number and status fields for data quality Initial value requests Recap requests Refresh messages  Opitional
  15. 15. Data Dictionary OpenMAMA supports self describing messages  Field type, name, id (integer) and value Sending names with every msg -> big messages Dictionary maps id’s to names, types and values  From a server as a message  Loaded from a file  Built dynamically Allows integration of non-self describing payloads
  16. 16. Fault Tolerance Applications can run in fault tolerant groups Every instance has a unique weight Heart beat messages determine failure  UDP multicast  Bridge using OpenMAMA and native middleware Callbacks inform instances of transitions Application responsible for recovery logic
  17. 17. OpenMAMA in Action
  18. 18. Let’s write a program! Create a publisher  Open a session  Pack a payload  Send message Create a subscriber  Open interest in a topic  Receive message callback  Unpack message
  19. 19. OpenMAMA Publisher Example Taken from int main(void) { mamapublisherc.c example initializeMama (); createIntervalTimer (); Publishes a message at createInboundSubscription (); regular interval createPublisher (); Subscribes to a request mama_start(gMamaBridge); topic return 0; } Sends reply on receiving a request Key concepts  Publishing  Subscribing  Replying to requests  Creating messages  Timers
  20. 20. Initialization void initializeMama(void) { Load middleware mama_status status; bridge status = mama_loadBridge(&gMamaBridge, gMiddleware);  By name “wmw”, if (status != MAMA_STATUS_OK) { printf ("Error loading bridge: %sn", “lbm”, “tibrv” or “avis” mamaStatus_stringForStatus (status)); exit (status); } Initialize OpenMAMA status = mama_open(); Create or get queue mama_getDefaultEventQueue(gMamaBridge, &gMamaDefaultQueue); Create a transport status = mamaTransport_allocate(&gTransport); status = mamaTransport_create(gTransport, • Two step } gTransportName, gMamaBridge); 1. Allocate 2. Create (initialize)
  21. 21. Create a Timer static void createIntervalTimer(void){ Initialize timer status = mamaTimer_create(&gTimer, gMamaDefaultQueue,  Default queue from timerCb, 1.0, initialization NULL); }  1 second interval static void MAMACALLTYPE  Repeating timerCb(mamaTimer timer, void* closure){ publishMessage (NULL); } Callback publishes message
  22. 22. Create a Subscription static void createInboundSubscription (void){ Subscription for gInboundCb.onCreate = createCb; requests gInboundCb.onError gInboundCb.onMsg = errorCb; = msgCb; gInboundCb.onQuality = NULL;  Send reply for each gInboundCb.onGap = NULL; gInboundCb.onRecapRequest = NULL; request mamaSubscription_allocate(&gSubscription); mamaSubscription_createBasic(gSubscription, Callbacks gTransport, gMamaDefaultQueue, gInboundCb, “REQUEST_TOPIC”, NULL);  No messages before } static void MAMACALLTYPE createCb inboundMsgCb(mamaSubscription subscription, mamaMsg msg, void* closure, void* itemClosure){ Two step create if (!mamaMsg_isFromInbox (msg)) { return; } msgCb for each } publishMessage (msg); message
  23. 23. Publishing Publisher requires transport and static void createPublisher(void){ mamaPublisher_create(&gPub, gTransport, topic. “MAMA_TOPIC”, NULL, NULL);  Optional source and root } static void publishMessage(mamaMsg request){ static int msgNumber = 0; Sending Messages static mamaMsg msg = NULL;  Static message reused if (msg == NULL) { mamaMsg_create(&msg); } else {  Standard MAMA fields: Status mamaMsg_clear(msg); and SeqNo } mamaMsg_addI32(msg, MamaFieldMsgStatus.mName, MamaFieldMsgStatus.mFid, MAMA_MSG_STATUS_OK);  Custom field: Publisher Topic mamaMsg_addI32(msg, MamaFieldSeqNum.mName, MamaFieldSeqNum.mFid, msgNumber++); mamaMsg_addString (msg, "PubTopic", 10002, “MAMA_TOPIC”);  request != NULL: called from if (request){ msgCb(). Point-to-point reply mamaPublisher_sendReplyToInbox(gPublisher, request, msg); } else {  Request == NULL: called from } mamaPublisher_send(gPublisher, msg); timerCb(). Multicast message. }
  24. 24. What Does it Do? Timer callback fired every second  Sends a message on “MAMA_TOPIC” Listens for messages on “REQUEST_TOPIC”  Message callback fired • If inbound message is a request then send reply
  25. 25. Middleware Bridges - Overview
  26. 26. Bridges Payload and Message Bridges  Directly reflect top-level API • Messages, subscriptions, queues, timers, etc.  One-to-one function mapping Virtual function table  Dynamic loading and static linking  Multiple bridges simultaneously  OpenMAMA macros build/load table • Bridge implementations just provide methods
  27. 27. mschonberg@nyx.com

×