This talk is split in two parts: - The first part describes how and why we end up using the Disruptor at ABC Arbitrage. - The second part is a catalogue of Disruptor specific patterns and practices.

1. OLIVIER
COANET
Le Disruptor à ABC Arbitrage :
une histoire, des patterns
PerfUG Paris 21/03/2018

2. A B C
abc-arbitrage.com
github.com/Abc-Arbitrage

3. Part 1
THE STORY

4. 2010
AUTOTRADING
FRAMEWORK
C++
C#
VS

5. 2011

6. 2011

7. 0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
1516
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Disruptor!
BY LMAX Exchange
2011

8. 0 1
2
3
4
5
6
7
8
9
10
11
12
13
14151617
18
19
20
21
22
23
24
25
26
27
28
29
30 31
Consumer 1
Consumer 2
Consumer 3
Head
2011

9. Hold on…
2011

10. Port
2011

11. Order Router
First .NET low latency
component
2012

12. NextGen
.NET trading platform
2014

13. .NET Disruptor maintainers
github.com/disruptor-net/Disruptor-net
2016

14. Boring
technology
2018

15. Part 2
PATTERNS

16. FRAMEWORK
LIBRARY
VS

17. A
D
G
I
B
H
C
E

18. A
D
G
I
B
H
C
E

19. FRAMEWORK!

20. Framework edge #1
Enforce a clean design

21. Framework edge #2
Single-threaded business
components

22. Framework edge #2
Single-threaded business
components
- Focus on business logic

23. Framework edge #2
Single-threaded business
components
- Focus on business logic
- Quant friendly

24. Framework edge #2
Single-threaded business
components
- Focus on business logic
- Quant friendly
- Deterministic behavior

25. Framework edge #3
Performance

26. +
EVENTS
EVENT HANDLERS

27. Disruptor < >T

28. XEngine
Let me introduce the

29. public enum XEventType
{
None,
MarketData,
Execution,
Acknowledgement,
TradingSignal1,
TradingSignal2,
// many other values...
}
The obvious enum
obvious enum is obvious

30. public class MarketDataInfo
{
public int SecurityId;
public long BidPrice;
public long AskPrice;
}
public class ExecutionInfo
{
public int SecurityId;
public long Price;
public long Quantity;
public string ExecutionCode;
}
// ...

31. public class XEvent
{
public XEventType EventType;
public MarketDataInfo MarketData;
public ExecutionInfo Execution;
public TradingSignal1Info TradingSignal1;
// ...
Option #1 Members for all event types

32. public void OnEvent(XEvent data, long seq, bool endOfBatch)
{
switch (data.EventType)
{
case XEventType.MarketData:
OnMarketData(ref data.MarketData);
break;
case XEventType.Execution:
OnExecution(ref data.Execution);
break;
// ...

33. Not as bad as it looks…

34. …up to a point

35. public class XEvent
{
public XEventType EventType;
public object EventData;
}
Option #2 Allocate or pool the event data

36. public void OnEvent(XEvent data, long seq, bool endOfBatch)
{
switch (data.EventType)
{
case XEventType.MarketData:
OnMarketData((MarketDataInfo)data.EventData);
break;
case XEventType.Execution:
OnExecution((ExecutionInfo)data.EventData);
break;
// ...

37. You just scrapped pre-
fetching and introduced
extra synchronization
Congratulations!

38. public class XEvent
{
public XEventType EventType;
public EventInfo EventData;
[StructLayout(LayoutKind.Explicit)]
public struct EventInfo
{
[FieldOffset(0)]
public ExecutionInfo Execution;
[FieldOffset(0)]
public MarketDataInfo MarketData;
}
Option #3 Unions

39. public void OnEvent(XEvent data, long seq, bool endOfBatch)
{
switch (data.EventType)
{
case XEventType.MarketData:
OnMarketData(ref data.EventData.MarketData);
break;
case XEventType.Execution:
OnExecution(ref data.EventData.Execution);
break;
// ...

40. Clean and packed…

41. …but requires caution

42. [StructLayout(LayoutKind.Explicit)]
public struct ExecutionInfo
{
[FieldOffset(0)]
public string ExecutionCode;
[FieldOffset(8)]
public int SecurityId;
[FieldOffset(12)]
public long Price;
[FieldOffset(20)]
public long Quantity;
}

43. [StructLayout(LayoutKind.Explicit)]
public struct MarketDataInfo
{
[FieldOffset(0)]
private object _padding0;
[FieldOffset(8)]
public int SecurityId;
[FieldOffset(12)]
public long BidPrice;
[FieldOffset(20)]
public long AskPrice;
}

44. …and .NET specific

45. public class XEvent
{
public XEventType EventType;
public ByteBuffer EventData;
// use a zero-copy wrapper like flatbuffer
// to read or write your event data
Option #4 LMAX style

46. Disruptor(Func<T> eventFactory,
int ringBufferSize,
TaskScheduler taskScheduler)
more choices…

47. int ringBufferSize

48. The disruptor is a
/BOUNDED/ queue

49. Unbounded queue aren’t

50. ringBuffer.Next()
ringBuffer.TryNext()
Backpressure
Retry / Drop

51. Keep queue sizes low

53. Pay attention to
backpressure locks

54. Use small ring buffer
sizes in test
environments

55. SHARED STATE

56. public interface IFxRateSource
{
NumericValue? GetFxRate(int currencyId);
}
public interface IFxRateManager : IFxRateSource
{
void UpdateFxRate(int currencyId, NumericValue value);
}
The shared type

57. The shared type
WRONG

58. IFxRateManager
Engine EEngine D
Engine C
Engine B
Engine A
Producer

59. Design principle #1
Stay away from any form
of synchronization

60. All state changes must
be the result of events
Design principle #2

61. Engine A Engine B Engine C
Enqueue multiple times
Store data locally in every consumer
FX rate producer
Engine A Engine B Engine C
FX rate producer
Engine A Engine B Engine C
FX rate producer
Engine A Engine B Engine C
FX rate producer
enqueue
enqueue
enqueue

62. Isolated Immutable Synchronized> >

63. CONFLATION
OPTIONS

64. 1
2
3
4
5
6
7
8
9
10
11
12
13
14
151617
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Event 9:
FX rate update
for currency A
Event 7:
FX rate update
for currency A
Consumer
Head

65. Conflation option #1
No conflation
YAGNI

66. Conflation option #2
Synchronization-based
NIH

67. private Dictionary<int, FxRateConflater> _conflaters;
public void Publish(int currencyId, FxRate fxRate)
{
_conflaters[currencyId].AddOrMerge(fxRate);
}
Producer side (1/3)

68. public void AddOrMerge(FxRate fxRate)
{
lock (_lock) // consider using a SpinLock
{
if (_currentEvent == null)
Add(fxRate);
else
Merge(fxRate);
}
}
Producer side (2/3)

69. private void Add(FxRate fxRate)
{
var sequence = _ringBuffer.Next();
_currentEvent = _ringBuffer[sequence];
_currentEvent.SetFxRate(_currencyId, this, fxRate);
_ringBuffer.Publish(sequence);
}
private void Merge(FxRate fxRate)
{
_currentEvent.FxRate.MergeWith(fxRate);
}
Producer side (3/3)

70. public void Detach()
{
lock (_lock) // consider using a SpinLock
{
_currentEvent = null;
}
}
Consumer side (1/1)

71. Conflation option #3
Batch-based
KISS

72. public void OnEvent(XEvent data, long seq, bool endOfBatch)
{
AddToBatchOrMerge(data);
if (endOfBatch)
{
ProcessBatch();
ClearBatch();
}
}
Consumer side (1/3)

73. private void AddToBatchOrMerge(XEvent data)
{
if (data.EventType == XEventType.FxRate)
{
var currencyId = data.FxRate.CurrencyId;
if (_fxEvents.TryGetValue(currencyId, out var previous))
{
Merge(ref data.FxRate, ref previous.FxRate);
return;
}
_fxEvents.Add(currencyId, data);
}
_batchEvents.Add(data);
}
Consumer side (2/3)

74. private void ProcessBatch()
{
foreach (var data in _batchEvents)
{
ProcessEvent(data);
}
}
private void ClearBatch()
{
_batchEvents.Clear();
_fxEvents.Clear();
}
Consumer side (3/3)

75. EVENT HANDLER
COMPOSITION

76. Handler 1 Critical
Handler 2 Critical
Handler 3 Non critical, fast
Handler 4 Non critical, fast
Handler 5 Non critical, slow

77. Logical
Physical
Design and
testing unit
Execution and
queueing unit

78. public class CompositeEventHandler<T> : IEventHandler<T>
{
private readonly List<IEventHandler<T>> _handlers;
// ...
public void OnEvent(T data, long seq, bool endOfBatch)
{
foreach (var handler in _handlers)
{
handler.OnEvent(data, sequence, endOfBatch);
}
}
}

79. Handler 1 Critical
+
Handler 2 Critical
Handler 3 Non critical, fast
+
Handler 4 Non critical, fast
Handler 5 Non critical, slow

80. BINARY LOGS

81. Binary log used as a
source of state
Journaling mode

82. Not so simple
Journaling mode

83. Binary logger handler
Business handler
Other handlers…
Journaling mode

84. Binary logger handler
Business handler
Other handlers…
Unsafe mode

85. Binary logs unusable as a
source of state, but…
Unsafe mode

86. An invaluable
debugging mechanism
Unsafe mode

87. A source of data for
business analysis
Unsafe mode

88. A source of data for
technical analysis
Unsafe mode

89. But requires
thoroughness
Unsafe mode

90. Conclusion

91. The end
OLIVIER
COANET
Thanks to @romainverdier and
@MendelMonteiro for the reviews