• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Build your own Service Bus V2
 

Build your own Service Bus V2

on

  • 602 views

Our feedback on the creation of a distributed enterprise service bus.

Our feedback on the creation of a distributed enterprise service bus.

Statistics

Views

Total Views
602
Views on SlideShare
602
Embed Views
0

Actions

Likes
1
Downloads
2
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Build your own Service Bus V2 Build your own Service Bus V2 Presentation Transcript

    • Build your own Service Bus ALT.NET Paris October 2013 ABC Arbitrage
    • Olivier Coanet OMG this guy does not tweet ABC Arbitrage Asset Management www.abc-arbitrage.com Kevin Lovato @alprema ALT.NET France www.altnetfr.org www.meetup.com/altnetfr
    • Build your own Service Bus
    • Application 1 Application 2 ESB Application 3 Copyright Bullshit Consulting ®
    • The tube is a lie
    • Application 1 Application 2 Qpid Service Bus V1: broker-based
    • Application Abc.ServiceBus Qpid.Client Qpid Bus library Broker-based Transport layer
    • Application 1 Abc.ServiceBus Qpid.Client Application 2 Abc.ServiceBus Qpid.Client Qpid + infrastructure services Qpid.Client Abc.ServiceBus Application 3 ESB
    • Service 1 Abc.ServiceBus Qpid.Client Service 2 Abc.ServiceBus Qpid.Client Qpid Qpid.Client Abc.ServiceBus Service 3
    • And they were blissfully, blissfully happy forever and ever
    • And they were blissfully, blissfully happy forever and ever
    • NEW REQUIREMENTS
    • Higher troughtput
    • Better latency
    • Custom messaging usage was missing Service Bus features
    • and the stack was kind of
    • version
    • LEFT DEAD .NET client
    • Buy or Build?
    • Brokerless transport recipe • Peer to peer communication protocol
    • Brokerless transport recipe • Peer directory
    • Brokerless transport recipe • Optional and parallel persistence
    • Brokerless transport recipe Chef tip: • Use interoperable ingredients
    • Brokerless transport recipe The cherry on the cake: • Build a gateway for Qpid communication
    • Maybe you want implementation details?
    • P2P communication Protocol Buffers
    • “ØMQ sockets are the world-saving superheroes of the networking world” Pieter Hintjens
    • Service 1 Service 2 Service 3 PUSH PUSH PUSH PULL Service 4
    • Who do we connect to?
    • Connecting to the directory Directory Service 1 Service 2 Service 3
    • Connecting to the directory Directory RegisterCommand + I listen to HelloEvent Service 1 Peers list + subscriptions Service 2 Service 3
    • Connecting to the directory Directory New service ID + subscriptions Service 1 Service 2 Service 3
    • Connecting to the directory Directory Service 1 Service 2 HelloEvent Service 3
    • The directory, our new SPOF
    • Connecting to the directories Directory 1 Service 1 Directory 2 Service 2 Service 3
    • Connecting to the directories Directory 1 Directory 2 RegisterCommand + I listen to HelloEvent Service 1 RegisterCommand + I listen to HelloEvent Service 2 Service 3
    • Connecting to the directories HelloEvent listeners: Service 1 HelloEvent listeners: Service 2 Directory 1 Service 1 Directory 2 Service 2 Service 3
    • Connecting to the directories HelloEvent listeners: Service 1 HelloEvent listeners: Service 2 Directory 1 Directory 2 RegisterCommand Service 1 Service 2 Peers list + subscriptions Service 3
    • Connecting to the directories HelloEvent listeners: Service 1 HelloEvent listeners: Service 2 Directory 1 Directory 2 HelloEvent listeners: Service 2 Service 1 Service 2 Service 3
    • Cassandra to the rescue Directory 1 Directory 2
    • What if a service goes down?
    • Persisting messages Service 1 Copy of HelloEvent Persistence service HelloEvent Stored messages Stored messages Stored messages Service 2
    • Persisting messages Service 1 Persistence service HelloEvent Ack Service 2
    • Persistence service • Persistence service acting as frontend • Cassandra for storage
    • Messages persistence model Row Key Abc_Service1 2013-09-03 11:03:52 Serialized message Abc_Service2 2013-10-03 08:25:38 … 2012-03-09 23:01:00 ... .. … …
    • Acks persistence model Row Key Abc_Service1 2013-09-03 11:03:52 null Abc_Service2 2013-10-03 08:25:38 … 2012-03-09 23:01:00 ... .. … …
    • Error recovery Service 1 Persistence service Stored messages Stored messages Stored messages Service 2 Messages sent to Service 2 while it was down
    • Error recovery • Phase 1 : Missed messages recovery Service 1 Persistence service Missed messages recovery Service 2 Stored messages Stored messages Stored messages
    • Error recovery • Phase 1 : Missed messages recovery • Phase 2 : Mixed recovery Service 1 Persistence service Normal message reception Streaming of the messages from the persistence Service 2
    • Error recovery Service 1 • Phase 1 : Missed messages recovery • Phase 2 : Mixed recovery • Phase 3 : Normal phase Persistence service Normal message reception Service 2
    • Meet our latest SPOF, the persistence
    • But actually, no • Persistance only required at startup • Messages are also stored in the ZMQ outbound buffer
    • Migration time !
    • Inter-bus gateway Service 1 Service 3 Broker Gateway Service 4 Zebus QPID Service 2
    • Summary • • • • Running in production Migration: 40% done Gateway under heavy load Owning the transport layer is double edged
    • Thank you