Equity Trading: Performance, Latency & Throughput
Prepared for the 2012 Exactpro ExTent Conference, Kostroma, Russia.
11° April 2012
Author:
Phil Penhaligan

Agenda
• Low Latency
– What are we measuring?
– How Fast Is Fast Enough?
– Who really needs it?
– What’s the next best thing?
– The Technology
• How did we get here?
• What next?
• Performance & Capacity Management
– The LSE/TQ KPIs
– System behaviour under extreme conditions
2

Low Latency: What are we measuring?
• 3 distinct attributes
– Measured from just outside LSE/TQ firewalls
– Excludes transmission time to/from external client/venue
– Excludes latency of client/venue’s own system(s)
– Excludes network packet retransmissions due to slow consumers
1. Order latency
• Time from Order receipt to order acknowledgment
• Commonly referred to as Order:Ack
2. Market Data latency
• Time from Order acknowledgement to public broadcast
Commonly referred to as Ack:Tick
3. Reference price latency (Turquoise Dark Book only)
• Time from Tick receipt to matching engine usage
3

Low Latency: What are we measuring?
• 2 key metrics
– Average
• Mathematical mean
– Consistency
• Max 99.9 percentile
– Discard the worst 0.1% of orders and then take the maximum
• Production Systems
– Continuous real-time capture
– Historical data stored in database for statistical analysis and reporting
• QA Systems
– New software releases evaluated during technical tests
– To confirm/verify expected latency improvements
– To detect unexpected latency regressions
4

Low Latency: Example – Turquoise Order:Ack
Latency Distribution Average and Consistency
Metric uS
Average 100% 108
Average 99.9% 103
Max 99.9% 381
Max 99 191
5

Low Latency: How Fast Is Fast Enough?
• Customer feedback
– LSE/TQ averages are already fast enough – better Consistency is most important!
• Our Focus
– Since the initial deployment of Millennium (Oct 2009 Turquoise and Feb 2010 LSE)
• We have improved order:ack average by 30% & consistency by 90%
• Multicast ack:tick has improved by 50%
– We continue to focus on improving consistency (see later slides for details)
• My Observations
– Since migration to Millennium IT, the exchange latency is an order of magnitude
better than latency within client applications + transmission to/from our venues
• Therefore for the majority of clients who are not co-located, further improvements to
latency will have little impact
• E.g. a 10% improvement to exchange latency = client 1% improvement
– Co-location customers whose algorithms interact exclusively with LSE/TQ markets
have the most to gain from further reductions in latency
– Our latency is a tiny fraction (<0.0005%) of human reaction times (typically <200mS)
6

Low Latency: Who really needs it?
• Clients/Participants
Category Least Sensitive Most Sensitive
Manual Trading Brokers, Arbitrage Traders,
“High Touch” Traders. Portfolio Traders.
Automated Trading Retail Service Providers (RSP), Market Makers.
DMA Clients.
Algorithmic engines & Remote Algorithms e.g. VWAP, POV, IS HFTs, Best Execution SORs,
Smart Order Routers. typically driven more by historical data Co-lo Algorithms (e.g. arbitrage,
curves than real-time ticks. momentum).
• The Venue/Exchange
– Venues with the lowest latency Market Data are at the front of
the queue when BBO updates occur
– Quality of dark book trades depends on ref price latency
– Builds strong technology reputation
– Generally good for Sales & marketing
7

Low Latency: What’s the next best thing?
• Consistency
– Fewer outliers, and outliers closer to the average rather than
simply seeking ever lower average latency is the most important
feature for any trading strategy whether manual or automated.
– If your system cannot compete on latency average, you can
always make improvements by making it more consistent.
• Resilience
– Closely tied to consistency is resilience, since an unresilient
system will never be able to provide consistent results!!
8

Low Latency: Technology
MIT/LSE/Turquoise Customers
How did we get here? • C++ • Fibre Optics
• Linux • Co-Location
• Infiniband • FPGA
• Native + ITCH
• 1Gb Extranex
What next? • Rebalancing • More automation
• More channels • Better Algorithms
• More partitions • Faster switches
• More threads • Faster firewalls
• FPGA + processors • FPGA + processors
• Tickerplant
• Lower level API
• Generation 8 …
• R&D
• Watch competitors!
9

Performance & Capacity Management – KPIs
# KPI Requirements / Target
1 Total Daily Transactions Max (4 x average, 2 x peak)
2 Total daily Trades Max (4 x average, 2 x peak)
3 Order Latency Agreed levels per market
4 Market Data Latency Agreed levels per market
5 Reference Price Latency Agreed levels per market (TQ only)
6 Transactions per second (1 sec peak) Max (4 x average, 2 x peak)
7 Transactions per second (10 sec average) Max (4 x average, 2 x peak)
8 Transactions per second (60 second Max (4 x average, 2 x peak)
average)
9 System Availability 100%
10

Performance & Capacity Management
• Each system is technically tested with every software
release to
– Prove KPI levels
– Reconfirm/prove behaviour and known breaking points
• Preliminary tests take place on pre-production
• At least one test cycle takes place on the actual
production hardware on a weekend
• Golden Rules
1. A client message will always get a valid response (ack or nack)
2. Any component that has been taken so far above the KPI level
that it fails must do so gracefully, and the system must
continue to obey golden rule 1.
• Because we know the system/component limitations &
bottlenecks we can manage growth as the markets
evolve
11

Performance & Capacity Management
Turquoise Lit Book Matching Engine Peak TPS last 12 months
12

To Summarise
• Low Latency
– What are we measuring? (1) Order:Ack, (2) Ack:Tick, (3) Ref Prices.
– How Fast Is Fast Enough? Millennium Average OK – Focus on Consistency.
– Who really needs it? Predominantly Market Makers, HFTs, Co-Lo Algos.
– What’s the next best thing? Consistency and Resilience.
– The Technology
• How did we get here? C++/Linux/Infiniband/Native/ITCH
• What next? Balancing/threading/FPGA/Tickerplant,
low level API/Generation 8 …
• Performance & Capacity Management
– The LSE/TQ KPIs Trans,Trades,latencies,TPS (1/10/60), Availability.
– System behaviour under 1. Always respond, 2. Fail gracefully.
extreme conditions
13

Questions?
14