Build your own

Service Bus

ALT.NET Paris

October 2013

ABC Arbitrage
Olivier Coanet
OMG this guy does not tweet

ABC Arbitrage Asset Management
www.abc-arbitrage.com

Kevin Lovato
@alprema

A...
Build your own

Service Bus
Application 1

Application 2
ESB

Application 3
Copyright Bullshit Consulting ®
The tube is a lie
Application 1

Application 2

Qpid

Service Bus V1:
broker-based
Application
Abc.ServiceBus
Qpid.Client

Qpid

Bus
library
Broker-based
Transport
layer
Application 1
Abc.ServiceBus
Qpid.Client

Application 2
Abc.ServiceBus
Qpid.Client
Qpid

+ infrastructure services

Qpid.C...
Service 1
Abc.ServiceBus
Qpid.Client

Service 2
Abc.ServiceBus
Qpid.Client
Qpid
Qpid.Client
Abc.ServiceBus
Service 3
And they were blissfully, blissfully happy
forever and ever
And they were blissfully, blissfully happy
forever and ever
NEW REQUIREMENTS
Higher troughtput
Better latency
Custom messaging usage was
missing Service Bus features
and the stack was kind of
version
LEFT

DEAD

.NET client
Buy or Build?
Brokerless transport recipe
• Peer to peer communication protocol
Brokerless transport recipe

• Peer directory
Brokerless transport recipe

• Optional and parallel persistence
Brokerless transport recipe

Chef tip:
• Use interoperable ingredients
Brokerless transport recipe

The cherry on the cake:
• Build a gateway for Qpid communication
Maybe you want implementation details?
P2P communication

Protocol Buffers
“ØMQ sockets are the world-saving
superheroes of the networking world”
Pieter Hintjens
Service 1

Service 2

Service 3

PUSH

PUSH

PUSH

PULL

Service 4
Who do we connect to?
Connecting to the directory
Directory

Service 1

Service 2

Service 3
Connecting to the directory
Directory
RegisterCommand
+ I listen to HelloEvent

Service 1

Peers list + subscriptions
Serv...
Connecting to the directory
Directory
New service ID + subscriptions

Service 1

Service 2

Service 3
Connecting to the directory
Directory

Service 1

Service 2
HelloEvent

Service 3
The directory, our new SPOF
Connecting to the directories
Directory 1

Service 1

Directory 2

Service 2

Service 3
Connecting to the directories
Directory 1

Directory 2

RegisterCommand
+ I listen to HelloEvent
Service 1

RegisterComman...
Connecting to the directories
HelloEvent
listeners:
Service 1

HelloEvent
listeners:
Service 2
Directory 1

Service 1

Dir...
Connecting to the directories
HelloEvent
listeners:
Service 1

HelloEvent
listeners:
Service 2
Directory 1

Directory 2

R...
Connecting to the directories
HelloEvent
listeners:
Service 1

HelloEvent
listeners:
Service 2
Directory 1

Directory 2
He...
Cassandra to the rescue

Directory 1

Directory 2
What if a service goes down?
Persisting messages
Service 1
Copy of HelloEvent
Persistence
service
HelloEvent
Stored messages
Stored messages
Stored mes...
Persisting messages
Service 1
Persistence
service

HelloEvent Ack
Service 2
Persistence service
• Persistence service acting as frontend
• Cassandra for storage
Messages persistence model
Row Key
Abc_Service1

2013-09-03 11:03:52

Serialized message
Abc_Service2

2013-10-03 08:25:38...
Acks persistence model
Row Key
Abc_Service1

2013-09-03 11:03:52

null
Abc_Service2

2013-10-03 08:25:38

…

2012-03-09 23...
Error recovery
Service 1
Persistence
service
Stored messages
Stored messages
Stored messages

Service 2

Messages sent to ...
Error recovery
• Phase 1 : Missed messages recovery
Service 1
Persistence
service
Missed messages recovery

Service 2

Sto...
Error recovery
• Phase 1 : Missed messages recovery
• Phase 2 : Mixed recovery
Service 1
Persistence
service

Normal messa...
Error recovery
Service 1

• Phase 1 : Missed messages recovery
• Phase 2 : Mixed recovery
• Phase 3 : Normal phase

Persis...
Meet our latest SPOF, the persistence
But actually, no
• Persistance only required at startup
• Messages are also stored in the ZMQ outbound buffer
Migration time !
Inter-bus gateway
Service 1
Service 3
Broker

Gateway
Service 4
Zebus

QPID

Service 2
Summary
•
•
•
•

Running in production
Migration: 40% done
Gateway under heavy load
Owning the transport layer is double e...
Thank you
Build your own Service Bus V2
Build your own Service Bus V2
Build your own Service Bus V2
Build your own Service Bus V2
Upcoming SlideShare
Loading in …5
×

Build your own Service Bus V2

1,173 views

Published on

Our feedback on the creation of a distributed enterprise service bus.

Published in: Business
  • Be the first to comment

Build your own Service Bus V2

  1. 1. Build your own Service Bus ALT.NET Paris October 2013 ABC Arbitrage
  2. 2. Olivier Coanet OMG this guy does not tweet ABC Arbitrage Asset Management www.abc-arbitrage.com Kevin Lovato @alprema ALT.NET France www.altnetfr.org www.meetup.com/altnetfr
  3. 3. Build your own Service Bus
  4. 4. Application 1 Application 2 ESB Application 3 Copyright Bullshit Consulting ®
  5. 5. The tube is a lie
  6. 6. Application 1 Application 2 Qpid Service Bus V1: broker-based
  7. 7. Application Abc.ServiceBus Qpid.Client Qpid Bus library Broker-based Transport layer
  8. 8. Application 1 Abc.ServiceBus Qpid.Client Application 2 Abc.ServiceBus Qpid.Client Qpid + infrastructure services Qpid.Client Abc.ServiceBus Application 3 ESB
  9. 9. Service 1 Abc.ServiceBus Qpid.Client Service 2 Abc.ServiceBus Qpid.Client Qpid Qpid.Client Abc.ServiceBus Service 3
  10. 10. And they were blissfully, blissfully happy forever and ever
  11. 11. And they were blissfully, blissfully happy forever and ever
  12. 12. NEW REQUIREMENTS
  13. 13. Higher troughtput
  14. 14. Better latency
  15. 15. Custom messaging usage was missing Service Bus features
  16. 16. and the stack was kind of
  17. 17. version
  18. 18. LEFT DEAD .NET client
  19. 19. Buy or Build?
  20. 20. Brokerless transport recipe • Peer to peer communication protocol
  21. 21. Brokerless transport recipe • Peer directory
  22. 22. Brokerless transport recipe • Optional and parallel persistence
  23. 23. Brokerless transport recipe Chef tip: • Use interoperable ingredients
  24. 24. Brokerless transport recipe The cherry on the cake: • Build a gateway for Qpid communication
  25. 25. Maybe you want implementation details?
  26. 26. P2P communication Protocol Buffers
  27. 27. “ØMQ sockets are the world-saving superheroes of the networking world” Pieter Hintjens
  28. 28. Service 1 Service 2 Service 3 PUSH PUSH PUSH PULL Service 4
  29. 29. Who do we connect to?
  30. 30. Connecting to the directory Directory Service 1 Service 2 Service 3
  31. 31. Connecting to the directory Directory RegisterCommand + I listen to HelloEvent Service 1 Peers list + subscriptions Service 2 Service 3
  32. 32. Connecting to the directory Directory New service ID + subscriptions Service 1 Service 2 Service 3
  33. 33. Connecting to the directory Directory Service 1 Service 2 HelloEvent Service 3
  34. 34. The directory, our new SPOF
  35. 35. Connecting to the directories Directory 1 Service 1 Directory 2 Service 2 Service 3
  36. 36. Connecting to the directories Directory 1 Directory 2 RegisterCommand + I listen to HelloEvent Service 1 RegisterCommand + I listen to HelloEvent Service 2 Service 3
  37. 37. Connecting to the directories HelloEvent listeners: Service 1 HelloEvent listeners: Service 2 Directory 1 Service 1 Directory 2 Service 2 Service 3
  38. 38. Connecting to the directories HelloEvent listeners: Service 1 HelloEvent listeners: Service 2 Directory 1 Directory 2 RegisterCommand Service 1 Service 2 Peers list + subscriptions Service 3
  39. 39. Connecting to the directories HelloEvent listeners: Service 1 HelloEvent listeners: Service 2 Directory 1 Directory 2 HelloEvent listeners: Service 2 Service 1 Service 2 Service 3
  40. 40. Cassandra to the rescue Directory 1 Directory 2
  41. 41. What if a service goes down?
  42. 42. Persisting messages Service 1 Copy of HelloEvent Persistence service HelloEvent Stored messages Stored messages Stored messages Service 2
  43. 43. Persisting messages Service 1 Persistence service HelloEvent Ack Service 2
  44. 44. Persistence service • Persistence service acting as frontend • Cassandra for storage
  45. 45. Messages persistence model Row Key Abc_Service1 2013-09-03 11:03:52 Serialized message Abc_Service2 2013-10-03 08:25:38 … 2012-03-09 23:01:00 ... .. … …
  46. 46. Acks persistence model Row Key Abc_Service1 2013-09-03 11:03:52 null Abc_Service2 2013-10-03 08:25:38 … 2012-03-09 23:01:00 ... .. … …
  47. 47. Error recovery Service 1 Persistence service Stored messages Stored messages Stored messages Service 2 Messages sent to Service 2 while it was down
  48. 48. Error recovery • Phase 1 : Missed messages recovery Service 1 Persistence service Missed messages recovery Service 2 Stored messages Stored messages Stored messages
  49. 49. Error recovery • Phase 1 : Missed messages recovery • Phase 2 : Mixed recovery Service 1 Persistence service Normal message reception Streaming of the messages from the persistence Service 2
  50. 50. Error recovery Service 1 • Phase 1 : Missed messages recovery • Phase 2 : Mixed recovery • Phase 3 : Normal phase Persistence service Normal message reception Service 2
  51. 51. Meet our latest SPOF, the persistence
  52. 52. But actually, no • Persistance only required at startup • Messages are also stored in the ZMQ outbound buffer
  53. 53. Migration time !
  54. 54. Inter-bus gateway Service 1 Service 3 Broker Gateway Service 4 Zebus QPID Service 2
  55. 55. Summary • • • • Running in production Migration: 40% done Gateway under heavy load Owning the transport layer is double edged
  56. 56. Thank you

×