Synchronization For High Frequency Trading Networks: A How To Guide
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Synchronization For High Frequency Trading Networks: A How To Guide

  • 1,267 views
Uploaded on

For many financial institutions, high frequency trading volume is growing at an accelerating pace and demanding new requirements on their IT infrastructure. Drivers in their business such as......

For many financial institutions, high frequency trading volume is growing at an accelerating pace and demanding new requirements on their IT infrastructure. Drivers in their business such as pricing of equities moving from decimal to penny resolution and the growing need for markets to provide improved liquidity are resulting in huge opportunities for financial gain. Taking advantage of these opportunities is, in part, dependent on the care taken in the network’s time synchronization and the management of latency. Wall Street firms who were involved in the early phases of High Frequency Trading have been early adopters of high performance timing solutions utilizing a variety of signals including GPS, IRIG, 1PPS, NTP and now the Precision Time Protocol (PTP) which allows for precision time transfer on Ethernet networks. The implementation of specific timing solutions depends on the trading infrastructure and the network topology. Through a combination of hardware, software, and careful network management, it is reasonable to expect microsecond level time-transfer from traceable time sources to Linux applications.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,267
On Slideshare
1,266
From Embeds
1
Number of Embeds
1

Actions

Shares
Downloads
8
Comments
0
Likes
0

Embeds 1

http://www.linkedin.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. White Paper:Synchronization for High Frequency Trading NetworksFor many financial institutions, high frequency trading volume is growing at an accelerating pace and demanding new requirementson their IT infrastructure. Drivers in their business such as pricing of equities moving from decimal to penny resolution and thegrowing need for markets to provide improved liquidity are resulting in huge opportunities for financial gain. Taking advantage ofthese opportunities is, in part, dependent on the care taken in the network’s time synchronization and the management of latency.Wall Street firms who were involved in the early phases of High Frequency Trading have been early adopters of high performancetiming solutions utilizing a variety of signals including GPS, IRIG, 1PPS, NTP and now the Precision Time Protocol (PTP) which allowsfor precision time transfer on Ethernet networks. The implementation of specific timing solutions depends on the tradinginfrastructure and the network topology. Through a combination of hardware, software, and careful network management, it isreasonable to expect microsecond level time-transfer from traceable time sources to Linux applications.IntroductionIT managers in financial services institutions, especially those who are participating in high frequency trading, find themselves facingmultiple converging and difficult to meet system-level requirements: • Surging network traffic in capital markets requiring accurate time-stamping of market data feeds which can approach a pace of over one million per second. • Processing data and making trading decisions in an ever more real time environment with high performance trading algorithms requires precision timing within the Linux kernel and software application. • Pressure to reduce trading latency and transaction times to sub-millisecond levels across a geographically disperse set of trading servers and exchanges. • Increasingly stringent regulatory oversight requiring high precision timed log files. • Ability to post-process nearly any network scenario for optimization.Spectracom is able to provide a timesynchronization solution for financial institutions:precision time sources referenced to worldwidetime standards, a variety of time distributionprotocols and standards, important precision timetransfer to trading server operating systems (theninto the application), verification solutions andportable clock calibrations where GPS is notavailable.The specific application of these products isdependent on the needs of the scenarios that arecommon for financial institutions now participatingin High Frequency Trading. However, there areseveral common approaches such as using GPS as atraceable time source, the new PTP protocol forprecise time transfer over a LAN, and sophisticatedclient software to improve precise time transfer tothe application within the Linux environment.Legally Traceable Time Through GPSAn important aspect of any time synchronizationdeployment is the notion of accurate, traceabletime, versus relative time. Time is absolute and oneof the seven legally-defined units of measure.www.spectracomcorp.com 1 | Timing & Synchronization White Paper
  • 2. Since 1972 the world’s time standard is UTC (coordinated universal time). It is based on atomic clocks maintained by NationalMetrology Institutes and is distributed in a number of ways. The most accurate and simplistic method (when afforded a clear view ofthe sky) of synchronizing to traceable time is through the GPS broadcast. Spectracom offers synchronization to GPS, in addition to anumber of other time sources, to transfer time to where it is needed in many different ways. Because of the myriad of choices oftiming references and timing signals, Spectracom offers SecureSync®, a network-based master clock that is feature-rich andconfigurable so you only pay for the functions that you need. Configured with a GPS receiver, SecureSync, like any other GPS-basedtiming hardware, offers better than 50 nanosecond accuracy to UTC. A good start for precision timing.A Combination of PTP Hardware and Software Improves SynchronizationLeveraging the network is a low-cost way to synchronize multiple servers (blades or 1U appliances) in the trading network. This hasbeen done for years through network time protocol (NTP). Today, a new network protocol is available to offer improved timetransfer over the network. Precision time protocol (PTP) uses hardware time-stamped messages to measure the delay between themaster and slave. The result is better accuracy when adjusting the slave clock for minimum offset compared to the master.A typical SecureSync configuration combines a GPS receiver and a PTP module. This “PTP grandmaster clock” is used to synchronizePTP slaves (clients) within the local area network with high accuracy when variation in bi-directional packet delay is low. Even underthe worst case scenario, PTP performance will be equivalent to NTP.An example of a PTP slave is the Spectracom TSync-PCIe-PTP time code processor card. The PCI express card has an integral LAN portto send and receive time-stamped PTP messages across the network to synchronize an on-board clock with 4 nanosecond resolution.This PTP slave offers the best possible time precision for a computer that can accommodate plug-in cards.Another example of a PTP slave is TimeKeeper Linux PTP client software. The software is a transparent and very lightweight “applet”that does not alter the Linux kernel and does not require modification of existing applications. It responds to the standard Linux timecalls such as gettimeofday. However it greatly improves the accuracy of timing functions in a number of ways: 1) Improves the accuracy of the local clock by disciplining it to the PTP time reference. 2) It quickly and smoothly converges local time to the master time with a typical offset less than 10 microseconds. 3) Time is monotonic and never goes backward which can occur with standard Linux system clocks.. 4) Time is computed without leaving the processor, therefore, is immune to interrupts and other operating system “overheard” that can delay time reads and affect timing accuracy. 5) Allows the application to build a high precision time-scale by making the current time available every 30-50 nanoseconds.Specialized time synchronization hardware improves TimeKeeper accuracy by a factor of 5. Therefore the best performance isachieved with a combination of TimeKeeper PTP client software and the TSync-PCIe-PTP card.The following describes the implementation of synchronization solutions in several scenarios using GPS, PTP, hardware masterclocks, PCI express cards, and timing management software. Also included are solutions for GPS-denied environments and NTPimprovement software.Typical Network ScenariosScenario I: Time synchronization within a local trading center, or between trading centers.Assumptions: • Timing packet delay variation (PDV) is within acceptable limits to produce the required time synchronization performance • GPS is available in all trading center locations • Most servers requiring synchronization are blade servers, however, some critical 1U servers require the highest level of synchronization • All servers are running Red Hat enterprise Linuxwww.spectracomcorp.com 2 | Timing & Synchronization White Paper
  • 3. Spectracom Solution: • Network Master Clocks: SecureSync® PTP Grandmaster (redundant pair) • Linux Time Client: TimeKeeper™ PTP Client • Local Hardware Clock/PTP Slave (for 1U application servers): TSync-PCIe-PTP plug-in card • Synchronization Hardware Testing: STA-61 Sync Tester/AnalyzerAs critical infrastructure, it is typical to deploy network master clocks as redundant pairs. Two SecureSync GPS PTP grandmasterclocks require their own GPS antenna (and RF cabling) with a clear view of the sky. PTP includes a “best master clock” algorithm sono special switches are required to accommodate a failover from one SecureSync to the other. Special care is required to minimizepacket delay variation (PDV) by eliminating as many indeterminate time delays in the network as possible and practical. Timingaccuracy is degraded due to timing packet path asymmetry. Use low traffic or directly connected PTP masters and slaves via hubssince switches and routers queue messages depending on network conditions. Alternatively, special PTP-enabled routers can bedeployed. Known as PTP transparent clocks, they are able to compensate for any delay of PTP messages.PTP slaves are deployed as PCI express cards for 1U application servers that can accept such hardware, and PTP Linux clients. Timingperformance across the network can be analyzed by measuring the phase of the 1 pulse-per-second signals from the PTPgrandmaster and the PTP hardware slave.This configuration can be duplicated across all data centers involved in the trading scenario. When each LAN is synchronized to atraceable time source via a local master clock, then any data passed from one LAN to another can be perfectly correlated –extremely important when managing latencies.Sidebar: Table 1 provides typical performance of a variety of time transfer protocols assuming minimal asymmetry between the PTPgrandmasters and slaves.Table 1: Time Synchronization Performance of GPS-based Time Transfer Technologies PTP w/ SW timestamps via TimeKeeper PTP w/HW timestamps via PCIe card NTP w/existing standard NTP Server alone and TimeKeeper 4-5 microseconds 1 microsecond 10 millisecondswww.spectracomcorp.com 3 | Timing & Synchronization White Paper
  • 4. Scenario II: Time synchronizing within a GPS-denied trading center.Assumptions: • Either the remote trading center is connected to another network with access to GPS and the variable path delay between networks is managed to minimize asymmetry or, • A procedure can be followed to calibrate a local master clock to GPS to keep it within the desired accuracy limitsSpectracom Solution: • Network Master Clock: SecureSync PTP Master/Slave • Linux Time Client: TimeKeeper PTP Client • Portable GPS Calibrator: GPS-12R Time-Transfer Standardwww.spectracomcorp.com 4 | Timing & Synchronization White Paper
  • 5. In the case of a network without access to GPS, there are 2 choices: 1) Manage the variable packet delay from a network with access to GPS (scenario 1) through the use of PTP-enabled infrastructure, or a directly connected dedicated low-traffic timing network, so a PTP slave can achieve accurate time from another network’s GPS PTP grandmaster. 2) Routinely calibrate a local PTP master using a “flying clock” portable calibrator.As previously mentioned asymmetric packet delay degrades the accuracy of PTP so it offers no advantage over the existing NTPinfrastructure. One approach is to manage the packet delay variation. A local SecureSync can be configured with two PTP modules;one to act as a PTP slave and one to act as a local PTP master for all the other clients on the network.If the PDV is unmanageable, then the local SecureSync should be configured as an “atomic clock” with a Rubidium oscillator toprovide the best timekeeping performance in the absence of continuous access to a traceable time source. (Configuring aSecureSync with a Rubidium oscillator is also a redundancy option in GPS-accessible environments to protect against failure of theGPS antenna system, RF cabling, or the entire GPS system itself.) The drift rate of a Rubidium oscillator is on the order of a part in 10 thto the 11 (<10 microseconds per day) and is several orders of magnitude better than crystal oscillators even when combined withtechniques to reduce the effect of temperature variation.While a Rubidium clock solves the “relative seconds” problem, traceability, or accuracy, to absolute time is required so events on thelocal network can be correlated to other networks for managing latencies, etc. Spectracom offers the model GPS-12R battery-powered GPS time-transfer standard to be used as a “flying clock”. Simply synchronize the unit in an area that can receive GPSsignals while connected to AC mains. Then transport it to the SecureSync unit under its internal battery power (2-3 hours, but a+12VDC option extends transportation time by using a vehicle battery). Then connect it to the SecureSync and reconnect mainspower. The SecureSync’s internal Rubidium oscillator will use the GPS-12R 1PPS signal as its primary reference. Once it issynchronized, disconnect the GPS-12R, and the SecureSync will resume in an accurate “holdover” condition until the next calibrationcycle.Sidebar: The following table below gives you the typical performance you will be able to achieve synchronizing remote locationsheld over with Rb oscillators. Table 2: Timing Performance between remote sites Between two sites both One site running on Between two sites both running GPS-based servers with Rb Oscillator running Rb servers with Timeframe servers constantly holdover Rb holdover 3 months 50 ns + PTP time transfer < 1 ms < 2 ms 6 months 50 ns + PTP time transfer < 2 ms < 4 ms 12 months 50 ns + PTP time transfer < 4 ms < 8 ms Perpetual 50 ns + PTP time transfer <10 us/day < 20 us/day Where: ms = milliseconds us = microseconds ns = nanosecondsScenario III: Optimizing Time Transfer to a Server RackAssumptions: • GPS is available in the trading center • A low traffic, highly symmetric, dedicated timing network can be established between the GPS master clock and the server rack or the network is PTP enabled • All servers are running Red Hat Enterprise Linuxwww.spectracomcorp.com 5 | Timing & Synchronization White Paper
  • 6. Spectracom Solution: • Network Master Clock: SecureSync® PTP Grandmaster • Server Rack Master Clock: SecureSync PTP Master/Slave • Linux Time Client: TimeKeeper PTP ClientSimilar to scenario II, a SecureSync GPS clock PTP grandmaster can accurately synchronize another SecureSync through a low PDVnetwork. This allows for physical separation of the GPS grandmaster and the server rack from a GPS accessible location to the “top-of-rack”. Network infrastructure can be used to transfer time via PTP. The top-of-rack SecureSync acts as a slave to the GPSgrandmaster and a master for all servers in the rack. The rack becomes a “mini timing network” to eliminate the PDV in the timetransfer between PTP master and client/slaves.Scenario IV: Improving NTP time synchronization within a local trading centerAssumptions: • NTP is preferred over PTP • There is a desire to improve the accuracy of time transfer • All servers are running Red Hat Enterprise LinuxSpectracom Solution: • Network Master Clock: Existing GPS NTP Server • Linux NTP Time Server: TimeKeeper NTP Server Software • Linux NTP Time Client: TimeKeeper NTP Time ClientNTP was designed for wide time distribution across many devices and networks in a cascading hierarchy. Accuracy of NTP timetransfer can be greatly improved by optimizing the algorithm for point-to-point synchronization. A version of TimeKeeper does justthat. Install this software application in any Linux server. Synchronization to a traceable time source is achieved from a timing signalthrough hardware such as a 1PPS signal from existing GPS timing hardware. The TimeKeeper NTP client synchronizes to the NTPserver in a similar way as the PTP client. TimeKeeper is offered in a combined NTP/PTP client, so a new PTP network can be overlaidon an NTP network. The same principals for managing packet delay variation apply to the NTP configuration as well as the PTPconfiguration.www.spectracomcorp.com 6 | Timing & Synchronization White Paper
  • 7. ConclusionFinancial institutions, hedge funds and market-makers need to address a variety of challenges with accurate, traceable time for theircritical networks and network elements. High frequency trading requires real-time decisions based on microsecond time accuracy inorder to remain competitive. This level of accuracy is available today through a combination of a GPS timing, new networksynchronization protocol (PTP), PTP masters and slaves, time-transfer Linux software, time calibration for GPS-denied environmentsand/or carefully managed networks, and measurement solutions.USA | 1565 Jefferson Road, Suite 460 | Rochester, NY 14623 | +1.585.321.5800 | sales@spectracomcorp.comFRANCE | 3 Avenue du Canada | 91974 Les Ulis, Cedex | +33 (0)1 64 53 39 80 | sales@spectracom.frUK | 6A Beechwood | Chineham Park | Basingstoke, Hants, RG24 8WA | +44 (0)1256 303630 | info@spectracom.co.uk June 8, 2011 - WP06-101 (A)www.spectracomcorp.com 7 | Timing & Synchronization White Paper