This document discusses non-functional testing approaches for financial markets software. It describes the structure of non-functional testing teams, how to prepare tests by configuring load injectors and defining load shapes, and the types of non-functional tests performed, including latency measurements, capacity tests, DLC testing, failover testing, and other approaches to evaluate system performance under stress conditions.

1. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com1
Non-functional testing approach
Open Access Quality Assurance & Related Software Development for Financial Markets
Tel: +7 495 640 2460, +1 415 830 38 49
www.exactpro.com

2. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com2
Contents
• NFT Team structure
• Tests preparation
• Test tools
• Latency measurements
• Capacity tests
• DLC test
• Failover tests
• Other NFT tests

3. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com3
NFT team structure

4. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com4
LSEG Non-Functional Testing Team

5. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com5
Test plan
Sample of an Non-Functional Test plan

6. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com6
Load Generators Classification
Measurement-Based Model-Based
Closed-Cycle Open-Cycle

7. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com7
Asymmetric Response Model

8. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com88
Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49
exactpro.com
Item Description
Exactpro Test Strategy
Step(s)
Load generation in trading systems
Testing Type Active Load and Non Functional Testing
Target SUT Trading Platforms, Market Data Delivery and Post-Trade Systems and their combinations
SUT Interface Back-end (typically connected to message gateways / APIs; data streams generation: mcast/ucast);
GUI Testing Capabilities not supported
SUT Interaction
Method
Inputs and outputs are generated based on the configured load shapes, parameters and templates.
Captured messages can be viewed and analyzed post-factum using the DB queries (Shsha) or/and
performance calculator tool (also developed by Exactpro)
Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, ITCH, etc.) and Proprietary (MIT, SAIL,
HSVF, RTF, RV, Quant House, etc.) protocols. New plug-ins for additional protocols developed by
request
Test Scripts Capable to stress the system with high rate of transactions including microbursts. Used for
Throughput, Bandwidth, Latency tests. Can be used for support of fault tolerance (Failover) tests
Test Management,
Execution and
Reporting
Simulation of multiple client connections with specified load shape for each connection or group of
connections (configure number of connections, messages templates, Load Shape for each connection
or group of connections, messages distribution for each connection or group of connections),
throughput up to 75,000 msg per core per second
Simulation of market data streams with required SLAs
Platform requirements Linux on 64-bit platform
Primary Competitor VeriFix TestPilot, HP Load Runner, IBM Rational Performance Tester, JMeter, Yandex.Tank
Load Injector test tool

9. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com9
Exchange testing common scheme

10. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com10
Tests preparation
• System configuration:
- Production setup (Instruments, trading parameters, etc)
• Data for building test load:

11. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com11
Tests preparation. Load injector configuration files
Load injector configuration files structure

12. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com12
Tests preparation
Load shape:

13. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com13
Latency results End-to-end
Do we actually send what we thought we send?
• Evaluation of message rate ‘per millisecond’ unit and order mix balance:
Number of msgs
per millisecond
% Samples
Inbound
(into MIT)
Outbound
(from MIT)
<5 55.64% 55.01%
5-8 3.67% 4.05%
8-10 2.60% 2.77%
10-15 5.32% 5.39%
15-20 5.88% 5.95%
20-80 26.85% 26.78%
>80 0.05% 0.05%
Partition 1
Message
Type
ME cores
Total0 1 2 3
Order 3.74% 3.02% 2.00% 4.14% 12.89%
Cancel 3.56% 2.89% 1.93% 4.02% 12.39%
Amend 0.60% 0.53% 0.34% 0.68% 2.16%
Quote 0.32% 0.11% 0.16% 0.27% 0.85%
Trades 0.24% 0.18% 0.13% 0.29% 0.84%
Order mix and load balance across ME cores:Message rate per millisecond:
• Internal system stats arbitration (rates per second and total amount of transactions)

14. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com14
Latency results End-to-end
%
Native
avg max
100 82 518
99.99 82 408
99.9 82 139
99 80 103
0
100
200
300
400
500
600
12
29
31
31
45
45
61
61
76
76
134
286
436
519
687
896
1299
2132
2545
3376
4874
8780
17074
21219
29643
35284
40723
29
29
Latency,us
rate, mps
max_100
max_99_9
max_99_0
0
50000
100000
150000
200000
250000
300000
350000
400000
450000
500000
1
6
11
16
21
26
31
36
41
46
51
56
61
66
71
76
81
86
91
96
101
106
111
116
121
126
131
136
141
146
151
156
161
166
171
176
Numberofsamples
Latency, us
Total
new_o
amend_o
new_q
cancel_o
Latency distribution:
Latency percentiles:

15. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com15
Other NFT tests
• Capacity test
– Check all external interfaces:
• Traiding FIX
• Trading Native
• PostTrade
• DropCopy
• ITCH Replay/Recovery
– Check hardware resources consumption by system components:
• CPU
• Memory
• Disk
• Network

16. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com16
Other NFT tests
• DLC test
– Pass system through Production like schedule:
• All trading cycles
• All scheduled sessions
– Apply appropriate load during various cycles to trigger extensions, special sessions, etc
– Perform some functional tests under load
– Data consistency check
• All expected messages are disseminated via ITCH during session transitions
• All Downstream systems connected and confirmed daily checks

17. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com17
Other NFT tests
• Failover tests
– Application failovers
– Hardware outages
• Application failovers
– Simulation of various outage types (SIGKILL, SIGSTOP, Stop via SysGuard)
– Failovers during different system state (at startup / trading day / EOD)
• Hardware failovers
– Network equipment failovers (Switches, Ports, NICs) – impact on Application
– Server isolations
– Oracle instances

18. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com18
Other NFT tests
• Rapid user actions tests (connect-disconnect, logon-logout)
– System should sustain against such user behavior
– HW resources consumption should not grow up
• Slow consumer tests
– System should handle such users and should has a protection against them
– HW resources consumption should not grow up
• Intensive usage of recovery channels
– System should be able to handle high number of requests on recovery channels and
should be able satisfy them
• Massive actions from Market Operations (mass order cancels, mass trade
cancels, mass instrument halts)
– System should handle Market operations’ actions like mass cancel of 10k active orders
or trades

19. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com19
Other NFT tests
TQ Special latency test – Main market Reference prices processing delay for TQ
dark book:

20. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49
www.exactpro.com20
Thank You!