1. Data Reconciliation and
Software Failures
Iosif Itkin, Anna-Maria Kriger
Exactpro Systems
ANALYSIS OF IMAGES, SOCIAL NETWORKS, AND TEXTS
April, 10-12th, Yekaterinburg

2. Exactpro Systems
Our company is a specialist firm focused on functional and non
functional testing of securities data distribution, trading systems,
risk management and post-trade infrastructures
Quality Assurance = Data Processing

3. Tools to Validate Trading Systems
ClearTH:
• Post-Trade testing tool
• Verifies each stage of the
DLC
• Integrated schedule
• Automated matrices
• Can create multiple days test
scenarios
• Concurrent multiple tests
• Integrated simulators
• SWIFT ISO protocol support
MiniRobots:
• Executes multithreaded java
code
• Complexity of test algorithms
is defined by the test developer
• Supports multiple client fix
connections, order entry and
market data via FIX
• Can use GUI to iterate
through sent and received
messages
Dolphin:
• Model-based
testing of market
surveillance systems
• Production-scale
capacity and
throughput
• Interactive real-time
alerts and reports
Shsha:
• Post-transactional tool
• Analyzes clients' activity and forecasts system
response
• Parses and displays logs in a user-friendly way
• Parses messages and then puts each to a data base
table where each column corresponds to each
message field
• Allows making summarized reports, etc
• Easy to understand GUI
Load Injector:
• Simulates multiple client connections with a specified load shape for each connection or a group of
connections
• Up to 75K messages / second from a single CPU core
• Measures latencies in microsecond range
• Performance test reports
Sailfish:
• Can test Order Entry, Market Data and Post
Trade connections in one test scenario
• Each test scenario is independent
• Allows running test scripts in any
sequence
• Simulation of multiple user connections
• Server simulators
• All messages are stored into a data base
• Generates test reports

4. Software Quality Assurance
Software quality assurance (SQA) is a process
that ensures that developed software complies with defined
or standardized quality specifications.
SQA is an ongoing process within the software development
life cycle (SDLC) that routinely checks the developed software
to ensure it meets desired quality measures.

5. Software Quality Assurance
Software quality assurance (SQA) is a process
that ensures that developed software complies with defined
or standardized quality specifications.
SQA is an ongoing process within the software development
life cycle (SDLC) that routinely checks the developed software
to ensure it meets desired quality measures.

6. Software Quality Assurance
Software quality assurance (SQA) is a process
that ensures that developed software complies with defined
or standardized quality specifications.
SQA is an ongoing process within the software development
life cycle (SDLC) that routinely checks the developed software
to ensure it meets desired quality measures.

7. Software Quality Assurance
Minimal Life & Health

8. Financial Services
Minimal Life & Health

9. Financial Services
Minimal Life & Health

10. Data Reconciliation
Reconciliation is a process of finding discrepancies in
data obtained from different sources.
In accounting, reconciliation refers to the process of
ensuring that two sets of records, usually account
balances, match each other.
In financial markets, data reconciliation systems help
asset managers to reconcile trades, cash and security
flows, balances and positions between different systems,
e.g. internal data stored by the trading participant vs.
external data received from counterparties, brokers,
clearers, custodians, etc.

11. Smart Order Router

12. Smart Order Router

13. Knight Capital Events
• 1 August 2012, USA
• Knight Capital – one of the most successful HFT firms
• Implemented changes related to Retail Liquidity
Program at NYSE
• SMARS – ultra-fast order router
• Source code responsible for legacy functionality
PowerPeg
• 212 parent orders, millions child orders
• Accumulated loss – $460m or $170k/sec
• Incorrectly configured risk systems
• Deployment on 7 servers instead of 8…

14. Knight Capital Events

15. Knight Capital Events

16. Knight Capital Stock Chart

17. Regulated Exchange

18. Facebook IPO on NASDAQ
• 18 May 2012, NASDAQ, One of the largest IPOs in history
• Secondary trading is preceded by a designate Display Only Period
(DOP)
• Multi-component architecture that included Matching Engine, IPO
Cross Application and Execution Application
• At the end of the DOP, NASDAQ’s “IPO Cross Application”
analyzes all of the buy and sell orders to determine the price at
which the largest number of shares will trade; then NASDAQ’s
matching engine matches the buy and sell orders at that price.
Usually takes 1-2 ms
• NASDAQ allowed orders to be cancelled at any time up until the
end of the DOP, including the very brief interval during which the
IPO cross price is calculated. After calculation was completed, the
system performed orders validation check between ME and “IPO
Cross Application”. If any of the orders were cancelled after the
start of the cross, the system would have to repeat the calculation

19. Facebook IPO on NASDAQ
• Over 496k orders participated in the cross, and its duration
exceeded 20ms
• Order cancellation arrived during this period, and the application
had to repeat the calculation. Two more cancellations arrived
during the second iteration, and four more - during the third one
• IPO Cross Application went into infinite loop at 11:05
• The NASDAQ team switched off validation check on the
secondary system and performed failover 25 minutes after the
start of the loop
• Unknown at that moment, 38k orders submitted between 11:11
and 11:30 were stuck and did not participate in the uncross. It
created another discrepancy, this time with Execution App and
Members who were not able to receive confirmation for orders
executed in the cross until 13:50

20. Failover Proposal

21. Failover Proposal

22. Facebook IPO Stock Chart

23. Questions and Answers
Thank you!
We look forward to seeing you there!