Trade and Event Processing at a Massive Scale - QCon NY 2012

1,810 views

Published on

Efficient, low-latency and in-order trade processing has always been at the focus of financial institutions and investment banks. In a world where every millisecond counts, parallelizing the processing as much as possible is essential. This presentation describes the common patterns for parallelism when handling events at massive scale

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,810
On SlideShare
0
From Embeds
0
Number of Embeds
363
Actions
Shares
0
Downloads
36
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • Algorithmic and high frequency trading
  • Much more complicated than this example: Exchange vs Market maker / trader checking the target market is open to take orderschecking the order is valid for that marketchoosing the right matching policy for the type of ordersequencing the order so that each order is matched at the best possible price and matched with the right liquiditycreating and publicizing the trades made as a consequence of the matchupdating prices based on the new trades
  • Much more complicated than this example: Exchange vs Market maker / trader checking the target market is open to take orderschecking the order is valid for that marketchoosing the right matching policy for the type of ordersequencing the order so that each order is matched at the best possible price and matched with the right liquiditycreating and publicizing the trades made as a consequence of the matchupdating prices based on the new trades
  • Much more complicated than this example: Exchange vs Market maker / trader checking the target market is open to take orderschecking the order is valid for that marketchoosing the right matching policy for the type of ordersequencing the order so that each order is matched at the best possible price and matched with the right liquiditycreating and publicizing the trades made as a consequence of the matchupdating prices based on the new trades
  • Much more complicated than this example: Exchange vs Market maker / trader checking the target market is open to take orderschecking the order is valid for that marketchoosing the right matching policy for the type of ordersequencing the order so that each order is matched at the best possible price and matched with the right liquiditycreating and publicizing the trades made as a consequence of the matchupdating prices based on the new trades
  • Much more complicated than this example: Exchange vs Market maker / trader checking the target market is open to take orderschecking the order is valid for that marketchoosing the right matching policy for the type of ordersequencing the order so that each order is matched at the best possible price and matched with the right liquiditycreating and publicizing the trades made as a consequence of the matchupdating prices based on the new trades
  • Much more complicated than this example: Exchange vs Market maker / trader checking the target market is open to take orderschecking the order is valid for that marketchoosing the right matching policy for the type of ordersequencing the order so that each order is matched at the best possible price and matched with the right liquiditycreating and publicizing the trades made as a consequence of the matchupdating prices based on the new trades
  • Much more complicated than this example: Exchange vs Market maker / trader checking the target market is open to take orderschecking the order is valid for that marketchoosing the right matching policy for the type of ordersequencing the order so that each order is matched at the best possible price and matched with the right liquiditycreating and publicizing the trades made as a consequence of the matchupdating prices based on the new trades
  • Mechanical sympathy
  • Two principles – data with message, staged processing Locking with every write/takeMechanical sympathy
  • Mechanical sympathy
  • Mechanical sympathy
  • Two principles – data with message, staged processing Locking with every write/takeMechanical sympathy
  • Mechanical sympathy
  • Mechanical sympathy
  • Mechanical sympathy
  • Trade and Event Processing at a Massive Scale - QCon NY 2012

    1. 1. http://www.nytimes.com/2011/08/27/business/as-trade-volumes-soar-exchanges-cash-in.html
    2. 2. http://www.nytimes.com/2011/08/27/business/as-trade-volumes-soar-exchanges-cash-in.html
    3. 3. http://www.bloomberg.com/news/2012-01-23/stock-trading-is-lowest-in-u-s-since-2008.html
    4. 4. http://www.tabbgroup.com/PublicationDetail.aspx?PublicationID=346
    5. 5. Buy Sell50, $12 60, $1060, $11 100, $1130, $10 30, $12
    6. 6. Price: $10Buy Sell50, $12 60, $1060, $11 100, $1130, $10 30, $12
    7. 7. Price: $10Buy Sell60, $11 10, $1030, $10 100, $11 30, $12
    8. 8. Price: $10Buy Sell60, $11 10, $1030, $10 100, $11 30, $12
    9. 9. Price: $10Buy Sell50, $11 100, $1130, $10 30, $12
    10. 10. Price: $11Buy Sell50, $11 100, $1130, $10 30, $12
    11. 11. Price: $11Buy Sell30, $10 50, $11 30, $12
    12. 12. Not Validated Validated Processed Validator Processor
    13. 13. Method Time in msec Single Thread 300 Single Thread w/ Lock 10.000 2 Threads w/ Lock 224.000 Single Thread w/ CAS 5.700 2 Threads w/ CAS 30.000 Single Thread w/ Volatile 4.700 Writehttp://disruptor.googlecode.com/files/Disruptor-1.0.pdf
    14. 14. Queue (Lack of) Fairness 50 Buy Sell 100 60Consumer Consumer Thread 1 Thread 2
    15. 15. Queue (Lack of) Fairness Buy Sell 50 100 60Consumer Consumer Thread 1 Thread 2
    16. 16. Queue (Lack of) Fairness Can you tell which order will be executed 1st? Buy Sell 100 60 50Consumer Consumer Thread 1 Thread 2
    17. 17. Validator Processor
    18. 18. Processor thread pool per segmentSymbol=A-H Symbol=I-S Symbol=T-ZValidator Processor Processor
    19. 19. Single Processor thread pool, pick random segment Symbol=A-H Symbol=I-S Symbol=T-Z Processor
    20. 20. Single processor thread per segment Symbol=A-H Symbol=I-S Symbol=T-Z Processor Processor Processor
    21. 21. Shared thread pool, mark segments under processing (CAS) Segment 1 Segment 2 Segment 3Segment 1Segment 2Segment 3 Processor
    22. 22. Shared thread pool, mark segments under processing (CAS) Segment 1 Segment 2 Segment 3Segment 1 XSegment 2Segment 3 Processor
    23. 23. Shared thread pool, mark segments under processing (CAS) Segment 1 Segment 2 Segment 3Segment 1 XSegment 2Segment 3 X Processor
    24. 24. Shared thread pool, mark segments under processing (CAS) Segment 1 Segment 2 Segment 3Segment 1Segment 2Segment 3 X Processor
    25. 25. Segments are created and removed as needed Processor
    26. 26. Segments are created and removed as needed “GOOG”“GOOG” Processor
    27. 27. Segments are created and removed as needed “GOOG”“GOOG” Processor
    28. 28. Segments are created and removed as needed “GOOG” “AAPL”GOOGAAPL Processor
    29. 29. Segments are created and removed as needed “GOOG” “AAPL” “AMZN”GOOG XAAPLAMZN Processor
    30. 30. http://martinfowler.com/articles/lmax.htmlhttp://www.nytimes.com/2011/08/27/business/as-trade-volumes-soar-exchanges-cash-in.htmlhttp://disruptor.googlecode.com/files/Disruptor-1.0.pdfhttp://www.gigaspaces.com/wiki/display/XAP9/FIFO+Grouping

    ×